Measure Before You Optimize
The single most expensive habit in performance work is proposing a fix before taking a reading. This is the loop that replaces it: Problem → Measure → Locate → Understand → Change → Measure Again — and the six questions that turn "it is slow" into a specific reading at a specific layer.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The loop
Performance work has a shape, and skipping steps is how teams spend a quarter making something 3% faster. Start from the problem as users experience it, not as the code presents it. Take a measurement that confirms the symptom exists and quantifies it. Locate where the time or resource actually accumulates. Understand the mechanism, because a fix built on a wrong mechanism works by accident or not at all. Change exactly one thing. Then measure again, against the same metric that defined the problem.
The last step is the one that gets dropped, and dropping it is what allows a team to accumulate a dozen "optimizations" while the p99 stays flat. An optimization is not complete when the code merges; it is complete when the production metric that defined the problem has moved, and you can show the before and after (see Regression or Tuesday? Telling a Real Change from Noise).
The loop is also a loop on purpose. Fixing the top bottleneck reveals the next one — a database that was hidden behind a CPU constraint, a lock that only shows up once throughput doubles. Expecting the bottleneck to move is what separates a performance *practice* from a one-off heroic fix (see The Bottleneck Moves After Every Fix).
The six questions
Every performance investigation can be opened with the same six questions. They are deliberately ordered: each one narrows the search space, and answering them out of order is how investigations wander.
The sixth question is the one that makes the difference between engineering and folklore. A change that improves a microbenchmark by 40% and the production p99 by nothing is not an improvement — it is a maintenance burden with a good story (see Microbenchmark or End-to-End: Why p99 Did Not Move).
| # | Question | What answers it | Why it comes here |
|---|---|---|---|
| 1 | What is the user-visible symptom? | The SLI: success rate, or latency at a percentile, for a named operation | If you cannot state it in user terms, you cannot tell whether you fixed it |
| 2 | Which signal confirms it? | A metric with a baseline: p99 of checkout_duration was 240ms, is now 1.8s | Confirms the problem is real and bounds when it started |
| 3 | At which layer does the time accumulate? | A trace waterfall, split by span (see Reading the Waterfall) | Narrows from "the system" to one component before any theorizing |
| 4 | Which resource is the constraint? | CPU, memory, disk, network, locks, pool, queue — via USE readings (see USE: Utilization, Saturation, Errors) | Decides which fixes are even physically capable of helping |
| 5 | Is it the average or the tail? | The distribution, not the mean (see The Average Was Fine and Users Were Not and Percentiles: Which One, and How Many Users Is That?) | Average and tail problems have different mechanisms and different fixes |
| 6 | Did the production metric move? | The same metric from question 2, after the change, over a comparable window | Without this, you have a story, not a result |
Why intuition points at the wrong code
The reason to measure is not that engineers are careless; it is that latency is distributed in ways human reading of code does not reveal. A function that looks expensive may run once per request and cost 2ms. A function that looks trivial may run 400 times because of an N+1 pattern invisible at the call site (see The Comb: N+1 as a Visible Shape). Time spent *waiting* — on a lock, a pool, a dependency — leaves no trace in the source at all.
There is also an arithmetic ceiling that intuition ignores. If a component accounts for 8% of request time, making it infinitely fast improves the request by 8%. The engineer who spends two weeks optimizing it is not wrong about the code being inefficient; they are wrong about it mattering. The budget view below makes that ceiling explicit before the work starts, which is why a latency budget is worth building early (see Latency Budgets: Spending 200 Milliseconds on Purpose).
The corollary is that the *first* measurement should be a breakdown, not a deep dive. Get the distribution of time across layers, then go deep in the one that dominates. Going deep first is how people end up with an extremely well-optimized 8%.
Key points
- Problem → Measure → Locate → Understand → Change → Measure Again. Skipping "measure again" is how teams accumulate optimizations without improvement.
- The six questions move from user symptom to production validation; answering them out of order is how investigations wander.
- If a component is 8% of request time, making it free improves the request by 8% — check the ceiling before starting the work.
- Time spent *waiting* — on pools, locks and dependencies — leaves no trace in the source, which is why reading code is not a substitute for a trace.
- Expect the bottleneck to move after every fix; that is the loop working, not a failure.
The Measure-First Loop
Change an input and watch which number moves — and which one does not.
What does the user actually experience? Not "CPU is high" — "checkout takes 6 seconds".
Starting from a resource dashboard means you will find a resource explanation, whether or not it is the cause.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Report → team: "the API is slow", with no operation, percentile or time window attached.
- 2Team → hypothesis: someone proposes a fix based on which code looks expensive, skipping the breakdown entirely.
- 3Change → production: the fix targets a component that accounts for a small share of request time, so the user-visible metric barely moves.
- 4Team → conclusion: the change is declared a success because a local benchmark improved, and the real bottleneck stays untouched.
- 5Next quarter → same symptom: the investigation restarts from zero because no baseline or breakdown was ever recorded.
- • "The profiler shows this function at the top, so optimizing it will fix the latency." A CPU profile ranks in-process CPU cost; if the request is dominated by waiting, the top frame may be irrelevant to latency (see Computing or Waiting?).
- • "The benchmark got 40% faster." A microbenchmark measures a component under conditions that may not resemble production traffic at all.
- • "Latency dropped after the deploy, so the fix worked." Traffic, cache state and time of day all move latency; a coincidence is not a validation (see Correlation Is Not the Root Cause).
- • "CPU is only 40%, so CPU is not the problem." Average utilization hides per-core saturation, run-queue depth and throttling.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Start with the user-facing SLI for the named operation, at a percentile, over a stated window — not a service-wide average.
- • Get a breakdown before a deep dive: a trace waterfall showing time per span, or a profile split by layer.
- • Record the baseline explicitly (metric, value, window) so the after-measurement has something to compare against.
- • Take the USE readings for the layer that dominates: utilization, saturation and errors on that specific resource.
- • Write the baseline down before changing anything: metric name, percentile, value, window, and the traffic conditions.
- • Take a breakdown first (trace or layered profile) and rank layers by share of total time; only then go deep.
- • Change one variable at a time so the after-measurement attributes cleanly.
- • Compare after against before on the *same* production metric over a comparable window and traffic level.
- • Record the result — including negative results — so the next person does not repeat a change that did not help.
- • The metric from question 2 must move materially at the same percentile, over a window with comparable traffic.
- • Confirm the improvement survives a peak period, not just a quiet one — a fix that only works at low load has not been validated.
- • Re-take the breakdown: the share of time in the changed layer should have dropped, and you should be able to name the new dominant layer.
- • Measuring first costs time during an incident, when pressure to "just try something" is highest — the discipline is genuinely uncomfortable.
- • Good baselines require telemetry that already exists; a team without it pays the instrumentation cost before the diagnosis cost.
- • One-variable-at-a-time is slower than shipping three changes at once, and it is the only way to attribute the result.
- • Add the SLI to a dashboard with a deploy marker overlay so a future regression is attributable (see "What Changed?" — Deploy Markers and the Invisible Deploys).
- • Encode the improved path in a load test or CI benchmark with a threshold, so a later change that undoes it fails loudly (see Regression or Tuesday? Telling a Real Change from Noise).
- • Keep the before/after record in the change description; a performance claim without a measurement is a claim nobody can re-check.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 1,200ms checkout breakdown is a teaching example chosen to show a realistic shape: waiting and external calls dominating, "obviously expensive" code trailing. Real breakdowns differ per system.
- WORKLOAD-SPECIFICWhich layer dominates depends entirely on the workload. A read-heavy cached API and a write-heavy transactional service have completely different breakdowns.