CPU or GPU for Inference
A workload decision: model size, available batch size, latency budget, cost per prediction and utilisation. A small tree model on CPU beats a GPU round-trip; a large transformer at volume does not fit on CPU. "GPU makes inference faster" is false as stated.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model is trained and needs hardware to serve it. Which questions decide whether an accelerator is worth its cost and its round-trip, and when does the CPU win outright?
A platform team standardised on GPU nodes for "all ML serving". Their busiest model is a gradient-boosted click predictor called a few thousand times a second; it now costs more than every other model combined, its p99 is worse than it was on the old CPU boxes, and the GPU utilisation graph is flat near zero.
GPUs are the ML hardware. Put every model on a GPU node: it is faster, it is what the training ran on, and one hardware class is simpler to operate than two.
The tree model's prediction is a chain of data-dependent branches, which is the workload a GPU is worst at; a CPU core finishes it before the input has crossed the bus to the accelerator (GPU Fundamentals).
- The tree model's prediction is a chain of data-dependent branches, which is the workload a GPU is worst at; a CPU core finishes it before the input has crossed the bus to the accelerator (GPU Fundamentals).
- The round-trip — serialise, copy over the interconnect, launch, copy back — costs more than the whole CPU prediction, so p99 got worse, not better.
- At a few thousand requests a second, one at a time, the GPU never sees a batch; utilisation is near zero and the bill is for an idle accelerator (Inference Batching, Inference Cost).
- The transformer, meanwhile, was underprovisioned because the GPU budget was spent on the click model; its tail latency is the one that actually needed the accelerator.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The click predictor estimates the probability a user clicks an item; the surrounding decision is which items to show. This lesson's target is the cost and latency of producing that estimate on a given hardware class.
- The properties traded are cost per prediction, latency per prediction and the utilisation of what is paid for.
- The click model is a tree ensemble with a few hundred trees over a few dozen features; one prediction is a few microseconds of branchy integer work on a CPU core.
- The team also serves a large transformer for content embeddings at lower volume, where a forward pass is a dense matrix workload that a CPU core takes tens of milliseconds to complete.
- Requests arrive individually with a tight budget for the click model and a looser one for embeddings.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A GPU delivers throughput on dense, regular arithmetic over large batches — matrix multiplies where thousands of identical operations run in lockstep. It is slow at branchy, irregular, latency-bound work, and every use pays a fixed cost to move data across the interconnect and launch kernels. A CPU core is fast at exactly the branchy, small, latency-bound work the GPU is bad at, and pays no transfer.
- So the question is the workload's shape. A tree ensemble over a few dozen features is branchy and tiny; it belongs on the CPU, where a prediction costs microseconds and the tail is set by the network, not the compute. A large dense model is a matrix workload; on a CPU it is tens of milliseconds and on a GPU with a batch it is a fraction of that per prediction.
- Batch size available decides whether the GPU's throughput can be used at all: without concurrent requests to group, the accelerator runs at batch size one and its advantage disappears into launch and transfer overhead (Inference Batching).
- Cost per prediction is the hardware cost per hour divided by predictions per hour at the utilisation achieved. An expensive accelerator at low utilisation loses to a cheap CPU at high utilisation even when the accelerator's peak throughput is far higher; quantization and compression move the boundary by shrinking the dense workload until it fits the CPU or fits more batches on the GPU (Quantization, Model Compression).
The shape of the workload, not the name of the field
The platform team's rule — ML serving runs on GPUs — treated the hardware as a property of the discipline. It is a property of the workload. The click predictor is a few hundred small trees: for each, follow a handful of data-dependent branches to a leaf and add a number. A CPU core does that in microseconds and the branch predictor learns the common paths. A GPU has to receive the features over the bus, launch a kernel that cannot do lockstep work on divergent branches, and send the result back; the transfer alone exceeds the whole CPU prediction.
The embedding model is the opposite shape: one large dense matrix multiply per input, the same operations on every element. That is what the accelerator was built for, and with a batch to fill it produces embeddings at a per-prediction cost the CPU cannot approach. The matrix scores both models on both classes; the caveat is that the scores are the argument's shape, not this fleet's measurements.
| Option | Latency | Cost | Operational | Note |
|---|---|---|---|---|
| Tree click model on CPU, in-process | Microseconds per prediction, no hop, cheap cores at high utilisation; the model's lifecycle is coupled to the caller. | |||
| Tree click model on GPU service | Transfer and launch dominate; batch size one at a few thousand requests a second; utilisation near zero on the most expensive hardware. | |||
| Transformer embeddings on CPU | Tens of milliseconds per input, a large core count to sustain volume; fine at low volume, the tail crosses the budget at peak. | |||
| Transformer embeddings on GPU, batched | Sub-millisecond per input inside a batch; needs a batcher, a warm-up and a fleet sized from the throughput curve. |
caveat The scores cannot say where the boundary falls for this fleet: that depends on the measured throughput curve, the batch size production traffic actually yields, the interconnect cost of the specific hosts and the hardware prices this quarter. A quantized transformer on CPU may sit between the last two rows and belongs in the comparison.
Five questions, asked per model
The decision is a short interview with the workload. How large and how dense is the model — is a prediction a matrix multiply or a walk down some trees? What batch size will serving actually see — is there concurrency to group, or do requests arrive one at a time? What is the per-prediction budget, including any hop and any queue wait? What is the cost per prediction on each class at the utilisation each will achieve? And, once deployed, what does the utilisation graph say?
The device gives the options with the conditions that select them. Two of the options are not hardware choices at all: shrink the model, or move it next to the caller. Both are often cheaper than either class.
For one model with its production traffic, which placement meets the budget at the lowest cost per prediction?
when Small or branchy model; tight per-prediction budget; requests arrive individually; the caller can carry the artifact.
cost Model lifecycle coupled to the caller's deploys and memory; a second consumer needs the same library and the same version.
when Small-to-medium model, several consumers, budget tolerates a hop; volume high enough to justify a fleet but not dense enough for an accelerator.
cost A service to operate; per-core throughput bounds the fleet size at peak.
when Dense model at a volume that fills batches inside the wait the budget allows; cost per prediction at achieved utilisation beats the CPU fleet.
cost Batcher, warm-up, transfer overhead, a fleet sized from the throughput curve, and the bill for any hour the batches do not form.
when A dense model whose budget is close on CPU: quantization or distillation may bring it under budget at a fraction of the accelerator cost.
cost Some quality, measured on the decision metric, and an evaluation to say how much.
What the bill and the graph are saying
Utilisation near zero on an accelerator is not an efficiency target missed; it is a diagnosis. The hardware is waiting on transfers, launches, or the arrival of a request that could be batched with the last one, and it is charging for the wait. The response is to match the workload to the class, not to search for traffic to keep the accelerator busy.
The assumption the placement makes is that the workload keeps its shape. A model swap from trees to a small network, a traffic change that brings concurrency, a hardware price change — any of them moves the boundary, and the standing measurement is what shows it moved.
The model's size and density, the batch size production traffic yields, and the cost per prediction at achieved utilisation are what they were when the hardware class was chosen.
holds when The same model family serves the same traffic pattern; cost per prediction and p99 are re-measured after any model or traffic change; utilisation on accelerator nodes stays near the batching knee.
breaks when The model is replaced by a different family; a client changes from bursts to a trickle or the reverse; hardware pricing or a quantized variant changes the cheapest class.
respond Re-run the five questions for that model, including the shrink-first option, and move it; do not tune a batcher or hunt for traffic to justify hardware the workload does not fit.
How to build it
Most important first.
- Decide per model, from five questions: how large and how dense is the model; what batch size will actually be available at serving time; what is the per-prediction latency budget; what is the cost per prediction on each hardware class at the utilisation each will achieve; and what does the utilisation graph look like once deployed.
- Default small and branchy models — trees, linear models, small networks over tabular features — to CPU, colocated with the caller where the budget is tight, so the round-trip disappears.
- Reserve accelerators for dense models at volume, behind a batcher, and size the fleet from the measured throughput curve rather than from request rate.
- Before buying the accelerator, try shrinking the workload: a quantized or distilled model on CPU often meets the budget at a fraction of the cost, and the comparison has to include that option (Pruning & Distillation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost per thousand predictions on each hardware class, at the batch size and utilisation actually achieved with production traffic. This is the number the decision turns on; peak throughput from a benchmark is not.
- End-to-end p99 per prediction including transfer and queue wait, against the budget. For the tree model the GPU loses on this alone.
- Not "GPU utilisation must be high" as a goal in itself. Low utilisation is the symptom that the hardware is wrong for the workload or the traffic; the response is to change the hardware, not to find traffic.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The workload shape that decided the hardware — model size and density, available batch size, arrival pattern — is still the production workload; a model swap or a traffic change can invalidate the choice.
- The measured cost per prediction and p99 on the chosen hardware were taken at production-shaped load, including transfer and queueing, and remain within budget at peak.
- The utilisation the cost estimate assumed is actually achieved; a fleet sized for a batch size that never forms is paying the unbatched price.
- Offline: benchmark each candidate hardware class with a replay of production traffic at production concurrency, measuring cost per thousand predictions and p99 including transfer; include the quantized and distilled variants in the comparison.
- Online: utilisation, batch-size distribution and cost per prediction per model per hardware class as standing dashboards; an alert on utilisation persistently near zero on accelerator nodes.
- Over time: re-run the comparison when a model is replaced, when the traffic pattern shifts, and when hardware prices change; the boundary moves with all three.
What can go wrong
- The comparison is run on a benchmark at batch size sixty-four and the GPU wins; production sends batch size one and the CPU would have won by a wide margin.
- The dense model is moved to CPU to save cost, and the tail latency at peak quietly crosses the budget on the days it matters, because the CPU comparison was done at average load.
- One hardware class is chosen for operational simplicity, and the cost of the wrong class for half the models is paid every month as the price of that simplicity.
- The click model is moved back to CPU but left behind a network hop to a separate serving fleet; the round-trip that was blamed on the GPU was mostly the hop.
- Two hardware classes are two things to operate, capacity-plan and debug; the simplicity of one class is a real benefit, and it has a price that should be stated per model.
- Colocating a CPU model with its caller removes the hop and couples the model's lifecycle to the caller's (Model Serving Architecture).
- Shrinking a dense model to fit the CPU costs some quality and an evaluation to say how much; the cost saving has to be weighed against that on the decision metric, not on a benchmark.
- "GPU automatically makes inference faster." For a small branchy model at batch size one it makes inference slower and far more expensive; the accelerator wins only on dense workloads with batches to fill. Faster is a property of the pairing, not the hardware.
- "Utilisation is low, so we need more traffic on the GPU." Utilisation is low because the workload does not fit the hardware. Moving the workload is cheaper than finding traffic.
- "We trained on GPU, so we serve on GPU." Training is a large-batch dense workload by construction; serving is whatever the traffic makes it. The two decisions are independent.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an accelerator wins on dense, batchable work and loses on small, branchy, latency-bound work — after paying a transfer — is a property of the hardware classes and holds regardless of framework; only where the boundary falls is workload-specific.
- MODEL-SPECIFICTree ensembles and linear models are branchy or tiny and belong on CPU at almost any volume; large transformers and convolutional networks are dense and belong on an accelerator once there are batches to fill — the same volume can point at opposite hardware for the two families.
- SCALE-SPECIFICAt low volume even a dense model may be cheaper on CPU because no accelerator is ever busy enough to amortise its cost; at very high volume the accelerator's throughput per unit cost dominates and the CPU fleet becomes the expensive option.
Where the depth lives
This domain teaches the model and hands the rest off by name.