Percentiles: Which One, and How Many Users Is That?
p50 describes the typical user, p99 describes the worst-served one percent, and the distance between them describes the system. The two errors that matter: reading a percentile without knowing the traffic volume behind it, and assuming percentiles compose across a call chain. They do not.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The ladder, and what each rung is for
p50 is the median: half of requests were faster. It describes the experience of a typical user and is the right number for "is the product generally fast". p90 and p95 describe the edge of normal — the experience of a user who hit a slightly unlucky combination. p99 and p99.9 describe the tail: users on cold caches, oversized accounts, degraded replicas, or the wrong side of a garbage collection pause.
Which rung matters depends on the question. Product quality perception tracks p50 and p90 most closely. SLOs are usually written at p95 or p99 because that is where "acceptable for nearly everyone" lives (see SLOs: A Target, a Window, and a Reason). Debugging a specific complaint means going past p99 into individual traces, because at that point you are looking for a request, not a statistic.
What no single rung tells you is the shape. Read them together: p50 47ms with p99 1,400ms is a completely different system from p50 88ms with p99 140ms, even though a naive "p99 is high" summary applies to the first only. The ratio between them is the fastest available read on distribution shape, as The Average Was Fine and Users Were Not works through in detail.
Traffic volume changes what a percentile means
A percentile is a position in a sorted list, so its statistical stability depends entirely on how long that list is. At 10 requests per second, a five-minute window holds 3,000 requests and p99 is determined by roughly the 30th-slowest — a handful of requests, any one of which can be a fluke. The number will swing wildly between windows with nothing changing in the system.
At 10,000 requests per second, the same window holds 3 million requests and p99 is determined by 30,000 of them. The number is stable, and more importantly it represents 30,000 real people having a bad time every five minutes. Identical statistic, completely different operational meaning.
This is the source of two opposite mistakes. On low-traffic endpoints, teams page on p99 and burn out on noise (see Alert Fatigue: The Page Nobody Reads); the fix is to alert on a longer window, a lower percentile, or an absolute count of slow requests instead. On high-traffic endpoints, teams dismiss p99 as "only one percent" — one percent of ten thousand per second is a hundred people per second, which is not a rounding error, it is a queue of support tickets forming in real time.
| Traffic | Requests in a 5-min window | Requests behind p99 | What the number means |
|---|---|---|---|
| 10 req/s | 3,000 | ~30 | Statistically noisy — will move on its own; alerting on it produces false pages |
| 500 req/s | 150,000 | ~1,500 | Stable enough to trust; 1,500 affected users per five minutes |
| 10,000 req/s | 3,000,000 | ~30,000 | Very stable; 100 users per second are getting the 1.2s experience |
Percentiles do not compose
This is the property that surprises people most. If a request calls five services in sequence and each has a p99 of 100ms, the request's p99 is not 500ms — and it is not 100ms either. There is no arithmetic that derives the composed percentile from the component percentiles, because it depends on whether the slow events are correlated.
The intuition that helps: if slowness is independent across the five services, a request needs to be unlucky only once to be slow, and the chance of hitting at least one slow dependency is much higher than 1%. That is the fan-out effect Fan-Out: Waiting for the Slowest of Seven covers, and it is why adding dependencies degrades the tail even when every dependency is individually healthy. If slowness is correlated — all five share a degraded database or a saturated network — the composed tail is worse still and moves in lockstep.
The operational consequence: measure the composed latency where the user experiences it, at the edge or in the client, and use per-dependency percentiles for attribution rather than for reconstruction. When the edge p99 is bad and every dependency p99 looks fine, that is not a contradiction to be explained away — it is the expected signature of fan-out, and the tool that resolves it is a trace with a critical path (see The Critical Path Is the Only Path That Pays).
1# Each dependency: 99% fast, 1% slow. Request waits for all five.2 3P(all five fast) = 0.99^5 = 0.9514P(at least one slow) = 1 - 0.951 = 0.049 # ~5% of requests, not 1%5 6# So the request-level p95 is already in "one dependency was slow" territory,7# and request-level p99 lands where TWO dependencies were slow.8 9# Correlated case (shared saturated DB): the five are not independent.10# P(at least one slow) collapses toward 1% -- but when it happens,11# all five are slow at once and the request is far worse than 100ms.12#13# Neither case is derivable from the component p99s alone.14# You must measure the composed latency directly.Key points
- p50 is the typical experience, p90/p95 the edge of normal, p99/p99.9 the tail — read them together, never one alone.
- A percentile without its traffic volume is uninterpretable: the same p99 can mean thirty noisy requests or thirty thousand affected users.
- p99 on a low-traffic endpoint is statistically unstable and a poor alerting signal; use a longer window or an absolute slow-request count.
- Percentiles do not compose across a call chain — five dependencies at p99 100ms do not produce a request p99 of 100ms or 500ms.
- Measure composed latency where the user experiences it; use per-dependency percentiles for attribution, not reconstruction.
Progressive depth
Overview
A percentile answers "how bad was it for the unluckiest N%". p50 is the typical user, p99 is the worst-served one in a hundred. Latency should always be described with at least two of them, because one number cannot distinguish a slow system from a system with a slow tail.
Practical
Read percentiles alongside request rate, and convert to affected users before deciding severity. On low-traffic endpoints prefer absolute slow-request counts over p99, which will otherwise swing on single requests and train everyone to ignore the alert.
Advanced
Percentiles do not compose. A request that waits on five independent 99%-fast dependencies is slow about 5% of the time, so the request-level tail is materially worse than any component tail. Correlated slowness changes the arithmetic again. Measure the composed value where the user is, and use component percentiles only for attribution.
Internals
A percentile from a bucketed histogram is an interpolation, so its precision is bounded by bucket width (see Histograms: A Distribution You Can Afford to Keep Forever); a percentile from a summary is exact for one instance and unmergeable across the fleet. Sketch encodings such as DDSketch provide relative-error guarantees across the whole range and merge correctly, which is why they are increasingly preferred for tail-sensitive work.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request → dependencies: one user request fans out to five services, each individually meeting a 100ms p99.
- 2Dependencies → composition: the request completes only when the slowest returns, so it inherits the union of five tails, not one.
- 3Composition → edge: edge p99 lands near 400ms even though no dependency ever reported worse than 100ms at p99.
- 4Edge → dashboard: per-service dashboards all show green; the edge dashboard shows a tail nobody can attribute.
- 5Dashboard → responder: without traces, the responder cycles through dependency dashboards finding nothing, because the effect exists only in the composition.
- • "p99 is only 1% of requests, it can wait" — at 10k req/s that is 100 people per second, continuously.
- • "p99 doubled overnight" — on a low-traffic endpoint this is frequently noise; check how many requests are actually behind the number.
- • "Every dependency is healthy, so the slowness must be in our code" — fan-out produces exactly this signature with healthy dependencies.
- • "We improved p99 by 40%" — check whether traffic fell over the same window; percentiles move when the population changes, not only when the system does.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Graph p50, p90 and p99 on one panel with the request rate visible on the same time axis.
- • Convert the percentile into people before deciding severity: `rate x window x (1 - percentile)` is the number of affected requests.
- • For low-traffic endpoints, prefer "count of requests slower than X" over a percentile — it is stable and directly interpretable.
- • When edge p99 is bad and dependency p99s are healthy, go to traces and read the critical path rather than re-checking the dependency metrics.
- • Alert on percentiles only where traffic makes them stable; elsewhere alert on absolute counts of slow requests.
- • Measure and SLO the composed, user-facing latency; keep dependency percentiles for attribution.
- • Reduce fan-out width or parallelize sequential dependency calls when the composed tail is the problem (see [[sequential-vs-parallel]]).
- • Add hedged requests or per-dependency timeouts when a single slow dependency dominates the composed tail (see [[timeouts-and-latency]]).
- • Confirm the composed edge p99 improved, not just individual dependency percentiles — the edge number is the one users have.
- • Check the affected-request count fell: percentile improvement with flat slow-request count usually means traffic changed.
- • Re-read p99/p50 to confirm the distribution narrowed rather than the whole distribution shifting.
- • High percentiles need high traffic to be stable; on small endpoints they generate noise that erodes trust in alerting.
- • Measuring composed latency at the edge tells you the truth but not the cause — attribution still requires tracing, which costs more.
- • Optimizing the tail (hedging, timeouts, retries) adds load and complexity, and can make the median slightly worse.
- • Bake the composed p99 into the SLO and its burn-rate alert (see Burn-Rate Alerts: How Fast Is the Budget Going?) rather than tracking per-dependency numbers.
- • Assert on p99 in load tests at production-like concurrency, where fan-out effects actually appear.
- • Alert when the number of dependencies on the critical path grows, since each one widens the composed tail.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEAll distributions, percentile values and traffic levels are constructed for teaching. The 0.99^5 calculation assumes independence, which real dependencies rarely satisfy — it is a lower bound on the effect, not a prediction.
- WORKLOAD-SPECIFICHow much traffic makes p99 stable depends on the distribution's shape and on how bursty the tail is. The thresholds shown are rules of thumb, not derived limits.