Latency Breakdown
A prediction request is parsing, feature fetch, preprocessing, model compute, postprocessing and network. The model is rarely the slow part for tabular systems, and almost always is for large networks.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The prediction endpoint is over its latency budget. Which of the six stages is actually consuming it, and at p50 or at p99?
A checkout page calls a fraud score before confirming the order. The endpoint's p99 has crept past the two-hundred-millisecond budget and the payments team wants the model "made faster". The model is a gradient-boosted ensemble that scores in under a millisecond.
The endpoint is slow, the model is the interesting component, so optimise the model: fewer trees, a smaller ensemble, maybe a compiled scorer. Report the mean latency before and after.
Model compute is a small fraction of the request even at p99. Halving it moves the p99 by a few milliseconds, and the budget is still blown.
- Model compute is a small fraction of the request even at p99. Halving it moves the p99 by a few milliseconds, and the budget is still blown.
- The tail is in the feature fetch: forty keys from a store in another zone, with an occasional slow key that takes the whole request with it. The mean hides this because most requests are fast.
- The team reported the mean and it improved. The payments team measured what they feel — the p99 — and it did not.
- Postprocessing turned out to include a synchronous call to a rules service that nobody remembered was in the path.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts chargeback risk at checkout; the serving system's target is a defined score within a fixed latency budget on every request, including the slow tail.
- The decision the latency serves is product-level: a slow score delays the confirmation page, and past the budget the caller times out and must act without a score.
- One request carries the order and the card token. The service fetches around forty customer and merchant aggregates from an online store, computes a dozen derived features, calls the model, applies a threshold and returns a decision.
- Request traces exist for a sample of traffic, with spans per stage. Nobody has looked at them by percentile.
- The online store is shared with two other services and sits in another availability zone.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A prediction request is a sequence of stages, each with its own latency distribution: parse the request, fetch features, preprocess into the model's input, run the model, postprocess the output into a decision, and the network on both ends. The request's latency is the sum, and its *tail* is dominated by whichever stage has the fattest tail, not the largest mean.
- For tabular models the feature fetch usually dominates: a network round trip per key or per batch of keys, to a store with its own tail. For large neural networks the model dominates: tens to hundreds of milliseconds of compute that scale with model size and sequence length (CPU or GPU for Inference).
- p50 and p99 describe different requests. The p50 is the typical path; the p99 is the request that hit a slow key, a garbage-collection pause, a cold cache or a retry. Optimising the typical path does not touch the tail, and the tail is what a caller's timeout sees.
- A budget is spent across stages. If the caller allows two hundred milliseconds and the network takes twenty each way, the service has one hundred sixty to spend, and every stage's p99 has to fit inside what is left after the others.
Six stages, two percentiles
Lay the request out as stages and measure each one twice — at the median and at the tail. The table is the whole diagnosis. The model is under a millisecond at both; the feature fetch is a modest median with a tail that consumes most of the budget; and a forgotten synchronous rules call sits in postprocessing.
The two columns tell different stories. At p50 the request is comfortably inside budget and the team's intuition that "the endpoint is fine" is correct for the typical request. At p99 the feature fetch alone is most of the budget, and it is the p99 the caller times out on.
| Stage | p50 (ms) | p99 (ms) | What drives the tail |
|---|---|---|---|
| Network in + parse | 2 | 6 | payload size, TLS resumption |
| Feature fetch (40 keys, cross-zone store) | 18 | 140 | one slow key, store GC, cross-zone hop |
| Preprocessing | 1 | 3 | allocation, a regex on a free-text field |
| Model compute (tree ensemble) | 0.6 | 1.2 | nothing that matters here |
| Postprocessing (threshold + rules call) | 4 | 35 | the synchronous rules service |
| Network out | 2 | 5 | response size |
Where the time goes depends on the model
The same six stages have a different profile for a large network. A transformer scoring a document does not need forty feature keys — its input is the document — but its compute is tens or hundreds of milliseconds and scales with input length. There the model is the endpoint, and the tools are batching, quantization and the accelerator.
This is why "make the model faster" is either the right instruction or an irrelevant one depending on which system you have, and why the breakdown has to be measured rather than assumed.
The endpoint is over budget at p99. Where does the trace say the time is?
when Tabular model, networked online store, many keys per request.
cost Batch the fetch, colocate the store, set per-fetch timeouts with defaults, cache stable features; accept a freshness question and a default-value path to monitor.
when Large neural network, long inputs, CPU inference.
cost Batch requests, quantize, move to an accelerator; accept a quality evaluation on slices for the quantized model and utilisation questions for the accelerator.
when A synchronous downstream call in pre- or postprocessing.
cost Remove it from the path or make it asynchronous; accept that the decision no longer waits for it.
when High concurrency, saturated workers.
cost This is a throughput problem — see Throughput vs Latency — and the fix is capacity or admission control, not any single stage.
The tail is a different set of requests
Percentiles are the vocabulary here and they are borrowed from performance engineering, where tail latency and its causes are their own subject. What the ML engineer adds is that the tail has ML-specific consequences: a request that times out at the feature fetch either fails the decision or scores on defaults, and both outcomes need a design (Serving Fallbacks).
The assumption the design rests on is that the stage allocation holds — the store stays inside its share, the model stays small. Both move without anyone deciding they should.
Each stage's p99 fits inside the share of the request budget the design allocated to it, so the sum stays under the caller's timeout.
holds when The store's p99 is stable and monitored per stage; model compute is a small share and model changes are latency-reviewed; no synchronous dependency has been added to the path.
breaks when The store degrades or gains a slow shard; a model change multiplies compute; a new feature adds a request-time call; traffic grows into the queueing regime.
respond Re-read the breakdown, fix the named stage, and re-derive the allocation; do not shrink the model on the assumption that the model is the problem.
How to build it
Most important first.
- Instrument every stage with a span and look at each stage's p50 and p99 separately, before changing anything. The stage with the largest p99 contribution is the target; it is usually not the model.
- Fetch features in one batched call rather than one call per key, colocate the online store with the service, and set a per-fetch timeout that returns a default for the missing keys instead of failing the request (Serving Fallbacks).
- Move work off the request path: precompute slow derived features, cache stable ones per entity with a short TTL, and drop any synchronous call that is not needed for the decision.
- Only then optimise the model, and only if its p99 share justifies it. For a large network that is the whole story and Quantization and Inference Batching are the tools; for a tree ensemble it is a rounding error.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-stage p99 as a share of the request p99 — the number that says where the budget goes. The mean is not a measurement of the problem the caller has.
- Fraction of requests that exceed the caller's timeout, and what the caller did about it. That is the product cost of the tail.
- Model compute time in isolation, on a benchmark, is the number most teams have and the one that least predicts the endpoint's latency.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The per-stage latency distributions observed in the trace sample are representative, including the tail; a sample that drops slow traces hides the problem.
- The feature store's p99 stays within the share of the budget the design allocated to it, and a degradation there shows up as a fallback-rate alert, not as silent defaults.
- The model's compute share remains small enough that model changes do not need latency review — an assumption that a growing model quietly breaks.
- Offline: replay a sample of production requests through the service with per-stage timing and plot each stage's distribution, not its mean.
- Online: alert on per-stage p99 rather than on the request p99, so the alert names the stage; track the default-value rate for the feature fetch alongside.
- Over time: re-read the breakdown after every model or feature change; the dominant stage moves.
What can go wrong
- Batching the feature fetch fixes the tail until the store's batch endpoint hits a slow shard, and now every request waits for the slowest of forty keys instead of an average.
- The per-fetch timeout returns defaults, and under a store degradation the model quietly scores everyone on defaults for an hour while the latency dashboard looks perfect.
- A model change doubles compute and nobody notices because the model was a small share; three changes later it is the largest stage and the trace has not been re-read.
- Batched fetches and colocation reduce the tail but couple the service to one store deployment; a per-key timeout with defaults bounds latency at the price of scoring on missing information.
- Precomputing and caching features improves latency and adds a freshness question (Feature Freshness) the request-time path did not have.
- Full per-stage tracing on every request costs overhead; sampling is cheaper and can miss exactly the slow requests you want to see unless the sampler keeps them.
- "The model takes a millisecond, so the endpoint should be fast." The model is one of six stages. The forty network reads before it are the endpoint.
- "Mean latency improved by a third, so the problem is fixed." The caller times out at the p99, and the p99 is a different set of requests from the ones that improved.
- "We need a GPU." For a tree ensemble whose compute is a small share of the tail, a GPU adds a transfer and changes nothing that matters. For a large network it may be the whole answer. The breakdown decides.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMULATEDThe millisecond figures in the breakdown table are illustrative of the shape — a tabular model behind a networked feature store — and were not measured on any real system; your trace will have different numbers and possibly a different dominant stage.
- MODEL-SPECIFICFor tree ensembles and linear models the feature fetch dominates and model compute is negligible; for transformer-scale networks the model is the dominant stage and the feature fetch is often trivial, so the optimisation target flips entirely.
- SCALE-SPECIFICAt low traffic the tail is mostly cold caches and GC pauses; at high traffic it is queueing in the store and in the model server, which is the subject of Throughput vs Latency.
Where the depth lives
This domain teaches the model and hands the rest off by name.