Latency Is a Distribution, Not a Number

"Latency is 120 ms" answers no question precisely. Latency of which operation? Measured at which point — the client's stopwatch, the load balancer, or the application handler after the request was dequeued? Over what window? And which part of the distribution: the typical request, the unlucky one, or the arithmetic mean that may describe neither?

The distribution below has a mean of 120 ms and a p50 of 62 ms. Most requests finish in well under a tenth of a second; a small population finishes in more than a second. The mean sits in the empty valley between the two humps — a value that almost no individual request actually experiences. This bimodal shape is extremely common in real services, and it usually means two code paths: cache hit versus cache miss, warm connection versus new TLS handshake, fast plan versus fallback plan.

So the first move when someone says "it is slow" is not to open the average. It is to ask *which requests* and pull the shape. If the shape is bimodal, the question changes from "why is the service slow" to "what puts a request in the slow hump", which is a far more answerable question — and one Percentiles: Which One, and How Many Users Is That? and Histograms: A Distribution You Can Afford to Keep Forever give you the vocabulary for.

The single most useful decomposition in performance work: response time = wait time + service time. Service time is the interval during which the request is being actively worked on. Wait time is everything else — sitting in a socket accept queue, waiting for a worker thread, waiting for a connection from the pool, waiting for a lock, waiting behind another request on the same shard.

This matters because the two respond to completely different fixes. If service time dominates, you make the work cheaper: a better algorithm, a better query plan, less serialization (Algorithmic Cost in a Request Handler, Self Time, Total Time, and Where the CPU Went). If wait time dominates, making the work cheaper barely moves the number — you need more servers, a bigger pool, less contention, or less arriving work. Teams that skip this split spend a quarter optimising a function that accounts for 8% of response time.

The tell is easy to read once you know to look: service time is roughly stable as load rises, while wait time grows with load, and grows non-linearly near saturation. If a p99 that was fine at 200 rps is terrible at 400 rps *with the same code*, you are looking at wait time and should read Queueing: Why Systems Get Slow Before They Get Broken next, not a profiler.

Latency accumulates along a path, and the path is longer than most mental models allow. Between the user pressing a button and the byte arriving, a request crosses the network, a load balancer, an accept queue, a worker pool, the handler, one or more datastores and usually an external dependency — and each of those boundaries is a place where waiting can be introduced without any code getting slower.

The waterfall below is the same 900 ms request seen as a trace. Note that the handler itself — the code someone would instinctively go optimise — is 40 ms of a 900 ms request. The 210 ms of pool wait and the 520 ms external call are the request. This is exactly the reasoning The Critical Path Is the Only Path That Pays formalises: shortening a span that is not on the critical path changes nothing a user can feel.

A practical habit follows: before proposing any fix, be able to state where the time went as a list that sums to the total. If you cannot produce that list, you do not yet have a diagnosis — you have a suspicion, and Measure Before You Optimize exists because suspicions are wrong often enough to be expensive.