GPU Fundamentals
A GPU is thousands of simple cores doing the same matrix arithmetic in lockstep. It is fast only when there is enough parallel work to fill it, which is why a single small request leaves it mostly idle.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
We moved inference to a GPU and per-request latency barely changed while cost went up. What is the GPU actually doing, and when does it help?
A support-ticket classifier — a small transformer — was moved from CPU instances to a GPU instance because "GPUs are faster for neural networks". Per-request latency dropped a little, GPU utilisation sits in single digits, and the monthly bill tripled. The platform team wants to know whether to move back.
A GPU is a faster processor for neural networks. Move the model to a GPU instance and inference gets faster; the framework handles the rest.
For one ticket the forward pass is a series of small matrix multiplications. Each one launches a kernel, the kernel finishes in microseconds, and the launch overhead and the host-device copies are a large fraction of the total. The device does almost no work per request.
- For one ticket the forward pass is a series of small matrix multiplications. Each one launches a kernel, the kernel finishes in microseconds, and the launch overhead and the host-device copies are a large fraction of the total. The device does almost no work per request.
- Latency improves modestly because the arithmetic is faster, but the fixed costs — launch, transfer, Python dispatch — did not move, and they dominate for a small input.
- Cost tripled because the instance is priced for a device that can do thousands of tickets per second, and it is doing three.
- Utilisation looks low and the team reads that as "headroom". It is starvation: the device is waiting for work it is never given in a shape it can use.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model assigns each incoming ticket to a queue. The serving system's target is a bounded per-ticket latency at the arrival rate, at the lowest sustainable cost.
- Tickets arrive one at a time, a few per second, and each is scored on arrival — the request pattern that a GPU is worst at.
- One request is one ticket of a few hundred tokens. The model is a few hundred megabytes of weights; a forward pass is a sequence of matrix multiplications whose size scales with token count.
- The serving process receives one request, tokenises it, copies the input to the device, runs the forward pass, copies the result back, and returns. No batching.
- The GPU utilisation metric reports the fraction of time any kernel is running; it does not report how full the device is during that time.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A GPU has thousands of arithmetic units organised so that one instruction is applied to many data elements at once — the same idea as CPU SIMD, at a much larger scale and with much higher memory bandwidth. It is built for workloads where the same operation runs over a large array: exactly a matrix multiplication.
- A forward pass is a chain of such operations. Each is dispatched to the device as a kernel; each kernel has a fixed launch cost, and the device reaches its throughput only when the kernel has enough independent work to occupy all its units for far longer than the launch. A batch of sixty-four tickets gives every matrix sixty-four times more rows and the same launch cost.
- Two things bound a kernel's speed: how fast the units can multiply (compute-bound) and how fast operands can be read from device memory (memory-bound). The ratio of arithmetic performed to bytes read — arithmetic intensity — decides which. Small batches have low intensity: the weights are read once per request and used for only a few rows, so the kernel is memory-bound and the arithmetic units idle (Memory Bandwidth & VRAM).
- Host-device transfers cross a bus that is far slower than device memory. For a small request, copying the input in and the result out can take longer than the compute; for a large batch it is amortised.
Wide, not fast
A CPU core is built to run one instruction stream as fast as possible: deep pipelines, branch prediction, large caches. A GPU is built to run the same instruction over thousands of data elements at once: many simple units, small caches per unit, and memory arranged to stream wide rows. A single scalar operation is *slower* on a GPU; a million identical ones are far faster.
A matrix multiplication is the ideal case. Every output element is an independent dot product, so the device can compute thousands at once, and a forward pass is mostly matrix multiplications. That is the entire reason neural networks and GPUs found each other — and the reason a forward pass over one short input, which produces small matrices, wastes the device.
Why a small batch starves the device
Each layer's kernel reads the layer's weights from device memory and multiplies them against the batch. With one ticket, the weights — megabytes — are read to produce a handful of output rows; the units finish almost immediately and wait for the next read. With sixty-four tickets, the same weights are read once and used sixty-four times as much. Arithmetic intensity rises with batch size, and the kernel moves from memory-bound toward compute-bound.
On top of that, each kernel launch costs a fixed few microseconds of host and device overhead. A forward pass is dozens to hundreds of kernels. At batch size one the launches can be a large share of the wall time; at batch sixty-four they are negligible.
Each layer reads all its weights to compute one row. The device spends most of the forward pass waiting on memory and on kernel launches. Occupancy is tiny; per-request latency is only modestly better than CPU; cost per prediction is the full device price divided by a trickle.
Each layer reads its weights once and computes sixty-four rows. Kernels are large enough to fill the units; launches amortise. Latency per request rises by the batch wait; throughput per device rises many-fold; cost per prediction falls accordingly.
The device's cost is fixed per hour and its speed is fixed per byte of weights read; the only way to spread both over more predictions is to give every weight read more rows to work on.
What must stay true for the GPU to earn its price
The decision to run on an accelerator rests on an assumption about traffic: enough concurrent work exists to keep the device full at an acceptable wait. That assumption can break in either direction — traffic falls and the device idles, or the input shape changes and batches no longer fit.
It is also the assumption the cost model in Inference Cost rests on. A GPU at low occupancy has the same hourly price as one at full occupancy and a fraction of the predictions per hour.
At the production arrival rate and wait cap, batches are large enough that kernel time dominates launch and transfer, and device occupancy stays high.
holds when Arrival rate times wait cap yields batches near the benchmark size; inputs are similar in length so batches pad little; the model stays resident on the device.
breaks when Traffic drops or becomes bursty; a product change makes inputs much shorter; the wait cap is tightened for latency; several small models share a device and each gets a trickle.
respond Consolidate traffic onto fewer devices, lengthen the wait cap on the asynchronous path, or move the low-traffic model back to CPU — the decision is per model and per traffic pattern.
How to build it
Most important first.
- Decide from the request pattern, not the model type. A steady stream of independent requests can be batched (Inference Batching) and a GPU becomes efficient; a trickle of latency-sensitive single requests often runs cheaper and about as fast on a CPU (CPU or GPU for Inference).
- If the GPU stays, batch at the model boundary with a wait cap, keep the model resident on the device, and pin inputs so the transfer is one copy rather than many small ones.
- Measure device occupancy during kernels, not the fraction of time kernels run. The utilisation metric says the device was busy; it does not say it was full.
- Benchmark the actual serving pattern — this arrival rate, this input length, this batch size — before choosing hardware. The vendor number is for a full device with a large batch.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Predictions per second per dollar at the required latency, measured with the real arrival pattern. This is the number that compares CPU and GPU deployments; per-request latency on an idle machine is not.
- Achieved device occupancy and memory-bandwidth utilisation during the forward pass; single-digit occupancy with a busy device says the kernels are too small.
- Per-request latency at the p50 on a quiet device is the number most benchmarks report and the one least related to cost.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The arrival rate is high enough, or requests tolerant enough of a wait cap, that batches fill and the device runs kernels large enough to amortise launch and transfer.
- The model and its working memory stay resident on the device between requests; reloading weights per request would put the bus in the request path.
- The benchmark that justified the hardware used the production request shape — input length, batch size, arrival pattern — and is re-run when any of those change.
- Offline: profile one request and one batch of sixty-four; compare time in kernels, in launches and in transfers. If kernels are a small share at batch size one, the device is starved.
- Online: dashboard occupancy alongside utilisation and cost per thousand predictions; compare against the CPU deployment on the same traffic.
- Over time: re-benchmark when the model changes size, the input length distribution shifts, or traffic changes shape.
What can go wrong
- Batching is added and the wait cap is longer than the CPU latency was; the GPU path is now more expensive *and* slower for the user, because the arrival rate never fills a batch.
- The model is small enough that the CPU's vector units were already close to the device's effective throughput at batch size one; the migration was a lateral move with a transfer added.
- Utilisation-based autoscaling reads the starved device as underused and scales the fleet down during a traffic surge, because the surge arrives as many small requests that never register as load.
- Batching to fill the device adds latency and a batching layer; without it the device is idle; with it the synchronous path pays a wait.
- A GPU instance is expensive per hour and cheap per prediction only when full; a CPU fleet is the reverse, and scales in smaller steps.
- Keeping models resident pins device memory per model, which limits how many models one device can serve and pushes toward a shared serving process with its own failure modes.
- "GPU automatically makes inference faster." It makes large, batched matrix arithmetic faster. For one small request most of the time is launch, transfer and dispatch, and none of that got faster.
- "Utilisation is low, so we have capacity to spare." Low utilisation on a starved device is waste, not reserve; the fix is bigger kernels, not fewer devices.
- "The model is a neural network, so it needs a GPU." It needs a GPU if the arithmetic per request, times the request rate, exceeds what a CPU's vector units can do at the latency budget. A small transformer at a few requests per second usually does not.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an accelerator needs enough parallel work to amortise fixed per-kernel costs holds for every GPU and TPU generation; the specific ratio of launch cost to kernel time varies by device and framework.
- SCALE-SPECIFICAt a few requests per second with small inputs a CPU is often cheaper and about as fast; at thousands per second, or with large models where one request is already a large matrix, the GPU wins by an order of magnitude and the CPU cannot keep up at any cost.
- SIMPLIFIEDThe compute-bound versus memory-bound distinction is presented at the level of arithmetic intensity without the roofline model or occupancy details, which the hardware domain covers; any throughput figures are for the shape of the argument.
Where the depth lives
This domain teaches the model and hands the rest off by name.