Inference Batching
Individual requests are grouped into a batch before the accelerator sees them. Throughput rises because the hardware runs one large matrix multiply instead of many small ones; latency rises because every request waits for the batch. Dynamic batching with a maximum wait is the knob.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The accelerator is idle most of the time and the per-request cost is high. How does grouping requests change throughput and latency, and where is the trade-off set?
A document-classification service runs a transformer on a GPU. Utilisation sits in the low single digits, cost per prediction is far above the estimate, and the team is asked to cut the GPU bill by half — without letting p99 latency cross the budget the upstream ingestion pipeline was built around.
Process each request as it arrives. The GPU is fast, the model is loaded, one request in and one prediction out is the simplest correct system, and adding queues only adds latency.
At batch size one the accelerator spends most of each request waiting on kernel launches and memory transfers, not computing; utilisation is a few percent, and the bill is for the idle time (GPU Fundamentals, Memory Bandwidth & VRAM).
- At batch size one the accelerator spends most of each request waiting on kernel launches and memory transfers, not computing; utilisation is a few percent, and the bill is for the idle time (GPU Fundamentals, Memory Bandwidth & VRAM).
- Throughput is bounded by sequential forward passes, so the service saturates at a request rate the hardware could handle many times over if requests arrived together, and the autoscaler adds GPUs to do more waiting.
- Cost per prediction is the GPU-hour divided by predictions served; at low utilisation it is dominated by the price of an idle accelerator (Inference Cost).
- Latency looks excellent on the dashboard at low load, which hides that the design has no headroom: the first burst queues at the network layer instead of at a batcher, with no maximum wait.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Classify each incoming document into a topic taxonomy; the label is the reviewed topic assignment. The target of this lesson is the cost and latency of producing that classification, not its quality.
- The consumer is a pipeline that submits documents as they arrive, at a rate that varies by an order of magnitude over the day.
- Requests arrive one document at a time, with bursts; each is a sequence of a few hundred tokens after tokenisation, padded to a fixed length.
- The model's per-request forward pass on the GPU takes about as long as a forward pass over a batch of dozens, because the hardware is latency-bound at batch size one.
- The latency budget is generous by online standards — well above a single forward pass — which is what makes batching possible.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An accelerator runs a matrix multiply over a batch in nearly the same time as over one row, up to the point where compute rather than memory movement or launch overhead dominates. Below that point, adding rows to the batch is close to free, so throughput scales with batch size while per-batch time barely moves.
- Batching therefore raises throughput and lowers cost per prediction; it raises latency because a request must wait for enough companions to form a batch, and then for the whole batch to finish. The wait is queueing latency, and it is the price of the throughput.
- Dynamic batching bounds the wait: the batcher collects requests until either the batch reaches a maximum size or a maximum wait elapses, then dispatches whatever it has. At high load batches fill quickly and the wait is short; at low load the wait hits the bound and the batch is small — latency stays under the bound and throughput adapts to demand.
- The trade-off is set by two numbers: maximum batch size, which caps memory and per-batch time, and maximum wait, which caps the added latency. Together with the arrival rate they determine utilisation and the tail; queueing theory, not intuition, gives the shape (Throughput vs Latency).
Why one large multiply beats many small ones
An accelerator is built to do a great deal of arithmetic per unit of memory moved and per kernel launched. At batch size one the weights are moved through memory for one row of input; the launch overhead and the memory traffic dominate and the arithmetic units idle. At batch size sixty-four the same weights are moved once for sixty-four rows, and the time hardly changes until the arithmetic finally becomes the bottleneck. That point is the knee.
The trade-off matrix below is the shape of the decision: three settings of the same service, scored on the axes batching moves. The caveat is the whole lesson — the numbers cannot say where the knee is on this hardware, or what the budget is.
| Option | Latency | Cost | Operational | Note |
|---|---|---|---|---|
| No batching (batch size 1) | Lowest per-request latency, lowest utilisation, highest cost per prediction; nothing to tune. | |||
| Dynamic, max wait a fraction of the budget | Fills at peak, dispatches small batches at low load; p99 stays under the budget if the wait was set from the tail, not the mean. | |||
| Dynamic, max batch at memory ceiling, long wait | Highest throughput, and the tail crosses the budget at peak when long sequences fill the batch; an out-of-memory failure takes out a whole batch. |
caveat The scores cannot express where the knee is for this model on this hardware with this sequence-length distribution, nor whether the latency budget makes any wait affordable at all; both have to be measured before the middle row is more than a guess.
Dynamic batching with a maximum wait
The batcher is a small loop: accept requests into a queue; when the queue reaches the maximum size or the oldest request has waited the maximum time, dispatch the batch to the model and resolve each request with its row of the output. At high load the size limit fires and the wait is short; at low load the time limit fires and the batch is whatever arrived. Latency is bounded above by the wait plus the per-batch time; throughput adapts to demand.
The two constants are the entire policy, and they should be derived, not guessed: the wait from the budget and the per-batch tail, the size from the measured knee. Validate before dispatch so one malformed request cannot fail its companions.
1MAX_BATCH = 32 # at the measured knee for this model + sequence lengths, not the memory ceiling2MAX_WAIT_S = 0.020 # budget - p99 per-batch time - headroom3 4async def batcher(queue, model):5 while True:6 first = await queue.get() # block until there is something to batch7 batch, deadline = [first], time.monotonic() + MAX_WAIT_S8 while len(batch) < MAX_BATCH:9 remaining = deadline - time.monotonic()10 if remaining <= 0:11 break # the oldest request has waited long enough12 try:13 batch.append(await asyncio.wait_for(queue.get(), remaining))14 except asyncio.TimeoutError:15 break16 xs = pad_to_longest([r.x for r in batch]) # bucket by length upstream, or this pads short docs to the longest17 ys = model(xs) # one forward pass; per-batch time nearly flat below the knee18 for r, y in zip(batch, ys):19 r.future.set_result(y)20 metrics.batch_size.observe(len(batch))21 metrics.queue_wait.observe(time.monotonic() - first.arrived)Watch the two histograms. Batches of size one at the maximum wait mean the arrival rate is too low to batch and the accelerator is the wrong hardware; queue wait near the maximum at peak means the size limit is too high for the budget.
The assumption batching makes about traffic
Batching assumes requests arrive close enough together to share a forward pass. That is a property of the traffic, and traffic changes: a pipeline that submitted documents in bursts is rewritten to trickle them; a load balancer is added that spreads requests over replicas so no batcher fills; the product's quiet hours grow. In each case the batcher keeps adding its wait and stops adding throughput, and the cost per prediction climbs back toward the unbatched figure while the dashboard shows the batcher "working".
The batch-size histogram is the monitor. When it collapses toward one, the trade has stopped paying and the decision is hardware, not tuning.
At the load levels where cost matters, enough requests reach the same batcher inside the maximum wait for batches to form near the knee.
holds when Arrival rate times maximum wait is comfortably above one at peak; routing sends a replica's worth of traffic to one batcher; bursts are not smoothed away upstream.
breaks when Traffic falls or is spread across more replicas than it can fill; a client starts pacing requests; the maximum wait is cut for a latency complaint without re-checking the arrival rate.
respond Consolidate batchers or replicas so batches can form; if the arrival rate is genuinely low, move to a smaller accelerator or CPU rather than tuning a batcher that has nothing to batch.
How to build it
Most important first.
- Put a batcher between the request handler and the model with a maximum batch size and a maximum wait, and set the wait from the latency budget: the budget minus the per-batch forward time minus headroom for the tail.
- Measure the throughput curve — predictions per second against batch size on the actual hardware and the actual sequence lengths — and choose the maximum batch size at the knee, not at the memory ceiling.
- Sort or bucket by sequence length where the model allows, so a batch is not padded to its longest member; a batch of short documents padded to one long one wastes most of the compute it was meant to save.
- Expose the batch size distribution and the queue wait as metrics; they are what say whether the batcher is doing anything, and they are the first thing to look at when the tail moves.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost per thousand predictions and accelerator utilisation, against the p99 of end-to-end latency. Those three together are the decision; any one alone is misleading.
- The distribution of dispatched batch sizes and of queue wait per request. A batcher whose batches are always size one at the maximum wait is adding latency and no throughput — the arrival rate is too low to batch, and the answer is a smaller accelerator or CPU (CPU or GPU for Inference).
- Not the model's forward-pass time. It is nearly constant across batch sizes, which is the whole point, and it says nothing about the queue in front of it.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The arrival rate is high enough, at the times that matter, for batches to form inside the maximum wait; otherwise the batcher adds latency and nothing else.
- Per-batch forward time at the maximum batch size, for the longest sequences that occur, fits inside the latency budget with the maximum wait added.
- Requests are routed so that a batcher sees enough of them — one batcher per accelerator, not one per request-handling thread.
- Offline: measure throughput and per-batch time against batch size and sequence length on the target hardware; replay a recorded day's arrival pattern through the batcher in simulation and read off p99 and utilisation for candidate settings.
- Online: batch size and queue wait histograms, utilisation and p99 by hour; a load test at peak shape with the longest sequences in the mix.
- Over time: re-measure when the model, the sequence length distribution or the arrival pattern changes — any of them moves the knee.
What can go wrong
- The maximum wait is set from the average forward time; at peak, batches are large and slow, the forward time grows past what the budget allowed for, and p99 crosses the budget precisely when throughput was needed.
- Batch size is set at the memory ceiling; a burst of long documents fills the batch with maximal sequences, the accelerator runs out of memory, and the whole batch fails — every request in it, not one.
- Two replicas each batch independently behind a load balancer that spreads requests evenly, so neither ever fills a batch; batching needs the requests to arrive at the same batcher.
- The batcher is placed after preprocessing that runs on the CPU per request, so the CPU becomes the bottleneck and the accelerator waits on tokenisation.
- Every request pays queue wait so that the fleet pays less; for a latency-critical path the wait may be unaffordable and the answer is a smaller model or no accelerator, not a shorter wait.
- A batch fails together: one malformed request can take out its companions unless the batcher validates before dispatch.
- Batching is only effective when requests reach the same batcher, which constrains load balancing and replica count in ways a stateless service would not have.
- "Batching makes inference faster." It makes the fleet do more per second; it makes each request slower by the wait. Whether that is a good trade is decided by the latency budget, not by the throughput number.
- "Set the batch size as large as memory allows." Past the knee, larger batches raise per-batch time and the tail with no throughput gain, and the memory ceiling is reached by the longest sequences, not the average.
- "Utilisation is low, so add more requests." If the arrival rate is genuinely low, no batcher fills, and the fix is a cheaper accelerator or CPU — not traffic the product does not have.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat grouping requests raises throughput at the cost of queue wait holds wherever per-batch cost is sublinear in batch size, which is true of accelerators and, to a lesser degree, of vectorised CPU inference; the numbers differ, the shape does not.
- MODEL-SPECIFICLarge neural models on accelerators are strongly sublinear and batching is the main cost lever; a small tree ensemble on CPU is nearly linear per row, and batching there buys little beyond amortising overhead.
- SIMPLIFIEDThe matrix and the queueing description treat per-batch time as flat up to a knee; on real hardware it rises gently before the knee and depends on sequence length and precision, and any figures implied are for the shape of the argument rather than measurements.
Where the depth lives
This domain teaches the model and hands the rest off by name.