Throughput: The Number That Means Nothing Without a Latency Bound
"We handle 10,000 requests per second" is not a capability claim until you say at what latency and with what error rate. Push a system to its maximum throughput and you will find the maximum is a place nobody wants to operate.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Throughput and latency are one measurement, not two
Every system has a curve. As offered load rises, throughput rises roughly linearly while latency stays roughly flat — the system is absorbing the work. Then comes the knee: throughput flattens because the bottleneck resource is fully committed, and latency starts climbing steeply because new arrivals wait behind existing work. Past the knee, throughput may even *fall*, as retries, timeouts and context switching consume capacity that used to do useful work.
The consequence is that "maximum throughput" is measured at the wrong place. The maximum is the top of the curve, which is deep in the region where latency is unacceptable and errors are appearing. The number you actually want is the maximum throughput at an acceptable latency, which sits before the knee and is often 50–70% of the peak figure. That is the gap between the 10,000 rps test result and the 6,000 rps production reality.
This is why a throughput claim without a latency bound is not a claim. State it as a triple: rate, percentile latency, error rate. "6,000 rps at p99 < 250 ms with errors under 0.1%" is a capability. "10,000 rps" is a marketing number.
| Offered load | Completed throughput | p99 latency | Errors | Reading |
|---|---|---|---|---|
| 2,000 rps | 2,000 rps | 85 ms | 0% | Linear region — plenty of headroom |
| 5,000 rps | 5,000 rps | 110 ms | 0% | Still linear; latency barely moved |
| 6,500 rps | 6,480 rps | 260 ms | 0.05% | Approaching the knee — latency is now rising faster than load |
| 8,000 rps | 7,700 rps | 1.4 s | 2.1% | Past the knee; timeouts starting |
| 10,000 rps | 8,100 rps | 6.2 s | 19% | "Peak throughput" — and completely unusable |
| 12,000 rps | 6,900 rps | 9.8 s | 41% | Congestion collapse: more load, less useful work |
Which unit, and per what?
Throughput is a rate of completed work, and "work" needs defining before the number means anything. Requests per second is the default for a web service, but it silently assumes requests are comparable — and if one endpoint serves a 2 KB JSON document and another exports 40 MB, an aggregate rps number describes nothing. The same trap appears with jobs per second across a queue with heterogeneous job types (Six Queue Signals, Two That Wake You Up).
Choose the unit that matches the bottleneck. If the constraint is CPU, work is roughly proportional to compute per request. If it is network egress, bytes per second is the real limit and rps is a proxy that breaks the moment payload sizes change (Payload Size: 20KB, 200KB, 5MB). If it is the database, queries per second or rows examined per second may be the honest unit. Getting this wrong produces capacity plans that are accurate right up until the traffic mix shifts.
A useful discipline is to record throughput alongside its composition — rps split by endpoint class, bytes per second, queries per second — so a capacity number can be re-derived when the mix changes rather than re-measured from scratch under incident pressure (Capacity Planning: Traffic to Machines).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| requests_per_second | 6,500 | Rate of completed work. On its own, says nothing about health. | normal |
| p99 request duration | 260 ms | Up from 110 ms at 5,000 rps — latency is rising faster than load. | suspect |
| CPU utilization | 78% | High but not pegged; utilization alone does not locate the knee. | suspect |
| DB pool waiters | 34 | Requests are queueing for a connection — the bottleneck has a name. | smoking gun |
| error rate | 0.05% | First timeouts appearing; past this point they compound. | suspect |
| bytes_out_per_second | 210 MB/s | Well under link capacity — network is not the constraint here. | normal |
More throughput and lower latency are often opposed
Several standard techniques raise throughput by *increasing* per-request latency, and the trade is usually worth making — as long as it is made knowingly. Batching amortises fixed costs across many items, raising items per second while every individual item waits for the batch to fill. Larger buffers keep a busy resource fed, improving utilization while lengthening the queue each request sits in. Higher concurrency limits admit more work, raising completions per second while every admitted request shares the same finite capacity.
The reverse trade exists too. Aggressive per-request latency optimisation — hedged requests, speculative work, aggressive prefetching — spends extra capacity to shorten individual requests, lowering maximum throughput in exchange for a better tail (Tail Latency: Why p50 Being Fine Does Not Help).
Neither direction is correct in general. What is not defensible is optimising one while reporting only the other, which is how a batching change ships as "40% throughput improvement" and arrives as a latency regression nobody predicted. Report both numbers for any change that touches this axis, which is exactly the discipline Every Optimization Buys Something and Sells Something is about.
- Batching — higher items/sec, higher per-item latency; the batch window is a latency floor you have chosen.
- Bigger buffers and queues — higher utilization of the bottleneck, longer waits for everything in the queue (Queueing: Why Systems Get Slow Before They Get Broken).
- Higher concurrency limits — more in flight, more sharing, worse tail; the limit is a latency control, not just a capacity control (Concurrency Limits: An Unbounded Server Is a Slower Server).
- Compression — fewer bytes per second on the wire, more CPU per request; helps when the network is the constraint and hurts when the CPU is.
- Hedging and speculation — better tail latency, lower maximum throughput because capacity does duplicate work.
Key points
- Sustainable throughput is the rate before the knee, not the peak rate — commonly 50–70% of the maximum a load test reports.
- A throughput claim needs a latency percentile and an error rate attached, or it describes an operating point nobody would accept.
- Past the knee, more offered load can produce *less* completed work as retries and timeouts consume capacity.
- Choose the throughput unit that matches the bottleneck resource: rps, bytes/sec, queries/sec, jobs/sec are not interchangeable.
- Batching, buffering and higher concurrency buy throughput with latency; hedging buys latency with throughput. Report both.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Load test → report: offered load is ramped to the point of maximum completions, recorded as "10,000 rps peak".
- 2Report → capacity plan: the peak figure is used for provisioning, with no latency or error bound attached.
- 3Production → users: at 6,000 rps p99 crosses 1 s and timeouts begin; the service is unusable well below its "capacity".
- 4Curve → responder: throughput and latency diverged at ~6,500 rps; everything above that was measuring the unusable region.
- 5Root cause → team: capacity was planned from the top of the curve rather than from the last point that met the latency objective.
- • "The load test proved we handle 10,000 rps" — it proved the system can complete 8,100 rps while 19% of requests fail and p99 is 6 s.
- • "CPU was only at 78%, so we had headroom" — utilization does not locate the knee; the constrained resource was the connection pool (Saturation: The Reading Utilization Cannot Give You).
- • "Throughput went down when we added load, so the load generator is broken" — falling completions under rising load is congestion collapse, and it is real.
- • "Requests per second is the capacity number" — only if requests are homogeneous. Mixed payload sizes make aggregate rps meaningless.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Run load in steps and plot completed throughput and p99 latency against offered load on one chart; the knee is where the curves diverge.
- • Record *completed* throughput, not offered load — past the knee they differ, and the difference is the story.
- • Capture the error rate at every step; a throughput figure measured while 19% of requests fail is not a throughput figure.
- • Split throughput by endpoint class and record bytes/sec alongside rps so the number survives a change in traffic mix.
- • Identify which resource stops scaling at the knee — pool waiters, CPU run queue, disk queue — because that names the bottleneck ([[saturation]]).
- • Define capacity as the throughput at which the latency SLO still holds, and provision against that number ([[capacity-planning]], [[headroom]]).
- • Add an explicit concurrency limit so offered load beyond capacity is shed quickly rather than absorbed into a growing queue ([[concurrency-limits]]).
- • Relieve the specific bottleneck the knee identifies — the pool, the CPU, the disk — rather than adding instances that share the same constraint.
- • Where throughput genuinely matters more than per-request latency, batch deliberately and state the new latency floor.
- • Re-run the stepped load test and confirm the knee moved to a higher rate, with the latency curve still flat at the previous knee.
- • Verify completed throughput and error rate together — a knee that moved because timeouts now fail faster is not a capacity improvement.
- • Check that the bottleneck moved to a different resource; if the same resource saturates at the same point, nothing changed ([[bottleneck-migration]]).
- • Operating well below the knee wastes capacity and money; the headroom is insurance you pay for continuously ([[headroom]]).
- • Concurrency limits mean shedding load — some users get a fast error instead of a slow success, which is a product decision.
- • Batching for throughput adds latency to every item and complicates failure handling, since a batch fails or retries as a unit.
- • Record the knee rate as a tracked number and re-measure it on a schedule, since it drifts with every dependency and code change.
- • Add a CI or staging load test that asserts the latency SLO holds at the planned operating rate (Regression or Tuesday? Telling a Real Change from Noise).
- • Alert on the ratio of current traffic to the known knee, so approaching capacity is visible before latency proves it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDThe load/latency table is derived from a single-bottleneck queueing model with a fixed service rate and retry amplification past saturation. The shape — linear region, knee, collapse — is general; the specific rates and latencies are not a measurement of anything.
- WORKLOAD-SPECIFICWhere the knee sits and how sharp it is depend on service-time variability, concurrency limits, timeout and retry policy. Two systems with identical peak throughput can have very different usable capacity.
Misconceptions
Apply it
Where the depth lives
The falling-throughput-under-rising-load regime was first characterised on the early internet, where retransmissions consumed the capacity they were competing for. The application-layer version — retries amplifying load on an overloaded service — is the same phenomenon at a different layer.