7 lessons

Load Testing & Benchmarking

Producing numbers that mean something: load-test shapes, coordinated omission, benchmark hygiene, microbenchmark versus end-to-end, and telling a real regression from noise.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

Load Testing: What Question Is This Test Answering?

Baseline, load, stress, spike and soak are five different tests answering five different questions. Most load tests fail before they start — against a warm cache, an empty database and one hot key, they measure a system that does not exist.

Symptom · The service passed its load test at 5,000 requests per second and fell over in production at 1,800. Everyone believed the number, and the number was measuring something else.
Load Test Shapes: The Shape Is the Hypothesis
▶ lab

Ramp, spike, step and soak are not stylistic choices. Each shape exposes a different failure: ramps find the knee, spikes find scaling lag and cold starts, soaks find leaks and drift. Choosing a shape is choosing what you are willing to find out.

Symptom · Every load test is the same steady rate for ten minutes, every load test passes, and production keeps failing in ways the tests never reproduced.
Coordinated Omission: When the Load Generator Lies

A load generator that waits for each response before sending the next one stops sending requests exactly when the system stalls. The requests that would have been slowest are never issued, never measured, and the reported p99 can be an order of magnitude better than what users experience.

Symptom · Load test percentiles look excellent and production percentiles for the same traffic level are far worse. The test is not obviously misconfigured, the request rate matches, and the numbers still disagree by a factor of ten.
Benchmarking: Does This Number Answer My Question?

A benchmark is an experiment, and most benchmarks fail as experiments before they fail as measurements. Warm-up, environment, workload realism, repetition, variance and a baseline are the difference between a number you can act on and a number you can quote.

Symptom · Two engineers benchmark the same change and get contradictory results. Both are confident, both ran the benchmark correctly, and nobody can say which number to believe.
Benchmark Fallacies: Confident Numbers That Are Wrong

Different environments, no warm-up, unrealistic payloads, averages without variance, several variables at once, and measuring something the system never actually does. Each produces a decisive number, and each is a reason to refuse to act on it.

Symptom · A benchmark result is circulating that justifies a large change. The number is specific, the chart is clean, and something about the setup does not survive a second look.
Microbenchmark or End-to-End: Why p99 Did Not Move

The function got 40% faster and the request did not. A microbenchmark measures one operation in isolation; an end-to-end benchmark measures the system with its contention, I/O and queueing. Each is misleading when asked the other's question.

Symptom · A well-measured optimization ships. The microbenchmark improvement is real and reproducible. The service's latency percentiles are indistinguishable from the week before.
Regression or Tuesday? Telling a Real Change from Noise

p95 moved from 180 ms to 260 ms. Before filing the bug, establish that both numbers answer the same question: same traffic mix, same data, same environment, enough samples. Then compare the difference against the noise you already know your measurement has.

Symptom · A dashboard or CI benchmark shows latency up 40% after a release. Half the team is convinced it is the release, the other half has seen this number move on its own, and nobody has the evidence to settle it.