Microbenchmark or End-to-End: Why p99 Did Not Move
The function got 40% faster and the request did not. A microbenchmark measures one operation in isolation; an end-to-end benchmark measures the system with its contention, I/O and queueing. Each is misleading when asked the other's question.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Two instruments, two questions
A microbenchmark isolates one operation, removes contention and I/O, warms everything, and measures it repeatedly. That isolation is the point — it produces a low-noise, high-resolution answer about that operation. It also means the number excludes everything that actually dominates a production request: waiting on a database, contending for a lock, queueing behind other work, serializing over a network.
An end-to-end benchmark includes all of it and pays for that realism with noise. The signal you are looking for might be a 3% shift buried in variance from cache states, neighbouring load and network jitter. It answers "did the system get better" and it answers it imprecisely.
Neither is better. Using a microbenchmark to predict system improvement is the error in this lesson; using an end-to-end benchmark to compare two hash functions is the mirror error, where the effect drowns in noise and you conclude there is no difference when there is a large one.
| Microbenchmark | End-to-end benchmark | |
|---|---|---|
| Answers | Is operation X faster? | Is the system faster for users? |
| Excludes | Contention, I/O, queueing, cache misses, network | Nothing — which is why the signal is noisy |
| Noise | Very low; small effects are detectable | High; small effects are undetectable |
| Fails when | Used to predict end-to-end improvement | Used to compare two small operations |
| Typical error | "40% faster function" → assumed 40% faster service | "No measurable difference" → concluded the change does nothing |
| Needs | Realistic inputs and warm-up | Production-scale data, realistic mix, many runs |
The arithmetic on the back of an envelope
The prediction is a single multiplication and it takes ten seconds. Find the operation's share of end-to-end request time from a profile or a trace, multiply by the fractional improvement, and that is your ceiling. The ceiling, not the estimate — contention, I/O and queueing will erode it further.
A function that is 4% of request time, made 40% faster, yields at most 1.6% end to end. That is invisible in production percentiles and completely consistent with a real, correctly measured microbenchmark win. The same 40% improvement on something that is 55% of request time yields 22%, which is a headline.
Doing this multiplication before the work is the highest-return habit in this module. It costs nothing, it selects targets, and it sets expectations so that "p99 did not move" is a prediction that came true rather than a mystery to investigate.
TARGET A — the function that was optimized share of request time 4 % microbenchmark improvement 40 % end-to-end ceiling 4 % x 40 % = 1.6 % p99 before / after 412 ms -> ~405 ms observable in production? no: below run-to-run noise TARGET B — the one the profile pointed at share of request time 55 % microbenchmark improvement 40 % end-to-end ceiling 55 % x 40 % = 22 % p99 before / after 412 ms -> ~321 ms observable in production? yes, unmistakably WHY "CEILING" AND NOT "ESTIMATE" The multiplication assumes the saved time disappears from the critical path. If the request then waits longer on a lock, a connection or a downstream call, some or all of the saving is absorbed and never reaches the user.
Using both, in the right order
The two instruments compose, and the order matters. Profile first, on production or production-like load, to find where request time actually goes (Self Time, Total Time, and Where the CPU Went). That gives you the share, which tells you whether the target is worth attention at all. Microbenchmark second, to iterate quickly on the hot operation with low noise and fast feedback. End-to-end last, to confirm the predicted improvement actually reaches the user.
Skipping the profile is how teams end up optimizing date formatting. Skipping the microbenchmark makes iteration painfully slow, because every experiment costs a full end-to-end run. Skipping the end-to-end confirmation is how a real microbenchmark win gets recorded as a service improvement it never delivered — and §162 applies here as everywhere: an optimization is not complete until measured where it matters.
When end-to-end does not move as predicted, that is information rather than failure. Either the share was wrong, or the saving was absorbed by something downstream — a lock, a pool, a dependency. The second case usually means the bottleneck moved rather than vanished, which is the next thing to go and find (The Bottleneck Moves After Every Fix).
Key points
- A microbenchmark answers "is this operation faster"; only a profile share turns that into a prediction about the request.
- End-to-end ceiling = share of request time × fractional improvement — and it is a ceiling, not an estimate.
- A 40% win on a 4% code path is 1.6% end to end, which is invisible in production percentiles and entirely expected.
- Order the work: profile to pick the target, microbenchmark to iterate, end-to-end to confirm it reached users.
- When end-to-end moves less than predicted, the saving was absorbed downstream — the bottleneck moved rather than disappeared.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Profile → share: the optimized function accounts for 4% of end-to-end request time.
- 2Microbenchmark → improvement: the function is 40% faster in isolation, correctly measured.
- 3Share × improvement → ceiling: 1.6% of request time is the best possible end-to-end outcome.
- 4Ceiling → observability: 1.6% of a 412 ms p99 is about 7 ms, well inside normal run-to-run variation.
- 5Observability → conclusion: p99 not moving is the predicted result, not evidence that the optimization failed.
- • "The optimization did not work" — it worked; its ceiling was below the noise floor and that was knowable in advance.
- • "End-to-end showed no difference, so the change is worthless" — for small operations, end-to-end is the wrong instrument.
- • "40% faster function means 40% faster endpoint" — only if that function were the entire request.
- • "The saving must be somewhere" — it may have been absorbed by a downstream wait, which is a different finding worth chasing.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Share of end-to-end request time for the target operation, from a CPU profile or a span in a trace.
- • Microbenchmark improvement as a fraction, measured with warm-up, repetition and realistic inputs.
- • Predicted ceiling, computed before the work starts, recorded so the outcome can be compared against it.
- • Actual end-to-end p50/p99 change at matched traffic after deployment.
- • Compute the ceiling before starting: profile share × expected improvement, and drop targets whose ceiling is not worth the effort.
- • Pick targets by share, not by how satisfying they are to optimize; the profile decides, not intuition.
- • Use microbenchmarks for iteration speed and end-to-end for confirmation, never one for the other's question.
- • Record the prediction alongside the change, so the post-deploy measurement is a check rather than a surprise.
- • When the saving is absorbed, profile again — the bottleneck has moved and the next target is different.
- • Compare the observed end-to-end change against the predicted ceiling; agreement validates both the profile and the change.
- • If the change is below the noise floor end to end, confirm it in an integration environment with reduced variance rather than declaring failure.
- • Re-profile after the change to confirm the target's share dropped as expected, which is measurable even when p99 is not.
- • Profiling before optimizing adds a step that feels like delay when the culprit seems obvious — and the obvious culprit is often 4%.
- • Maintaining both microbenchmarks and end-to-end benchmarks doubles the test surface to keep current.
- • Optimizing strictly by profile share can neglect tail-latency causes that consume little average time but dominate p99.
- • Keep the microbenchmark in CI to protect the optimization, and keep the profile share current to know if it still matters.
- • Record predicted versus actual end-to-end improvement for each optimization; a systematic gap means the profile is not representative.
- • Re-run the profile after significant changes, since shares shift and yesterday's hot path may no longer be one.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 4% and 55% shares, the 40% improvement and the resulting p99 figures are teaching arithmetic. The multiplication is exact; the inputs are invented.
- WORKLOAD-SPECIFICProfile shares vary by route, by input distribution and by load level. A share measured under light load can be very different at peak, when contention and queueing dominate.