Performancebenchmarkingmeasurementdead codewarmuptimer

Every Way a CPU Microbenchmark Lies

A microbenchmark measures what it measures, which is frequently not what you meant. The compiler deletes work whose result is unused, the caches and predictors are warm in ways production never is, the clock speed moves underneath you, and the timer itself costs more than the operation. Each of these has produced published results that were simply wrong.

Follow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
Why does my microbenchmark say this code is fast when production says otherwise?
What you wrote
The loop runs the operation ten million times and divides the elapsed time by ten million, which gives the cost of one operation.
What the hardware does
The compiler may have removed the loop entirely, the branch predictor learned the pattern after a few hundred iterations, the cache is holding a working set production never sees, and the core boosted to a clock it cannot sustain.
Microbenchmarks are how most hardware-level performance beliefs get formed, and an unreliable one produces a confident wrong belief that propagates through a codebase for years.
Source→Compiler→Instructions→Front End→Execution→Registers→Caches→Memory→I/O→Behavior

The compiler deleted your benchmark

The first and most common failure is that the work never happened. A compiler is permitted to remove any computation whose result is unobservable, and a benchmark loop computing values nobody reads is exactly that. The symptom is a result that is impossibly good — an operation apparently taking a fraction of a cycle, or a loop whose time does not scale with its trip count.

The reliable defence is to make the result observable in a way the compiler cannot see through: accumulate into a value that is eventually consumed, write to a volatile location, or use whichever compiler-specific "do not optimise this away" primitive the benchmarking library provides. Merely assigning to a local variable is not enough, because the compiler can see that the local dies unused.

The correct check is arithmetic rather than faith: scale the trip count and verify that the time scales with it. A loop that takes the same time for a million and ten million iterations did not do the work. This single check catches the majority of deleted-benchmark cases and costs nothing.

Measures nothing — result unused, loop removable
1start = now();
2for (i = 0; i < N; ++i) {
3    result = expensive(data[i]);   # result never used
4}
5elapsed = now() - start;
6# Reports an impossible per-op cost. The loop may not exist
7# in the emitted code at all.
Result is observable; the work must happen
1start = now();
2acc = 0;
3for (i = 0; i < N; ++i) {
4    acc ^= expensive(data[i]);     # feeds a consumed value
5}
6elapsed = now() - start;
7consume(acc);                      # opaque to the optimiser
8 
9# And verify: double N, confirm elapsed roughly doubles.

The accumulator makes each result observable, so the computation cannot be removed. The trip-count scaling check is the independent confirmation that it was not removed anyway.

Everything in a microbenchmark is warm

A benchmark loop is the most predictable code the machine will ever see. After a few hundred iterations the branch predictor has learned every branch, the instruction cache holds the entire loop, the data cache holds a working set sized for the benchmark rather than for production, and the TLB has every page it needs. The measured cost is therefore the *steady-state, fully warmed, best possible* cost — which is a real number, and often not the relevant one.

Production is the opposite in every respect. The branch is unpredictable because the data varies, the working set is far larger and shares cache with everything else in the process, the code path is entered from many different callers, and pages are cold. A function measured at a few nanoseconds in isolation can cost far more in situ, entirely without either measurement being wrong.

The second-order version of this catches people who have already learned the first: a benchmark that measures the same input repeatedly measures a cached result path. Benchmarks over sorted or uniform data measure a branch predictor operating at its best. If the production distribution is not the benchmark distribution, the benchmark is answering a question nobody asked — the theme Benchmark Fallacies: Confident Numbers That Are Wrong develops from the systems side.

What a microbenchmark holds constant that production does not
ResourceIn the benchmarkIn productionConsequence
Branch predictorFully trained on a repeating patternData-dependent, frequently mispredictingBenchmark hides misprediction cost entirely
Instruction cacheHolds the whole loopCompeting with the rest of the call graphFront-end stalls absent from the measurement
Data cacheBenchmark-sized working set, residentLarger, shared, evicted between callsMiss cost systematically understated
TLBFew pages, all mappedMany pages, sparse accessTranslation cost invisible
ClockBoosted; short burstSustained; thermally limitedBenchmark runs at a frequency production cannot hold
NeighboursMachine otherwise idleSharing caches, bandwidth and coresContention entirely unmeasured

The clock moved, and the timer is not free

MICROARCH-SPECIFICBoost behaviour, timer instruction cost, and sensitivity to code alignment all vary by vendor and generation; alignment effects in particular are tied to specific front-end structures that differ between designs.

Two further effects act on the measurement itself rather than on the code. The first is frequency: a short benchmark runs in a boost state the machine cannot sustain, so it reports a throughput that will not survive contact with a sustained workload. The fix is to run long enough to reach steady state, and to report that steady state rather than the first burst — the argument The Clock Is a Variable and The First Ten Seconds Lie make in detail.

The second is timer overhead and resolution. Reading a clock costs cycles, and if the operation being measured is of comparable cost, the measurement is mostly measuring the clock. The standard fix is to time a batch of many operations and divide, which amortises the timer cost — but that reintroduces the warm-cache and trained-predictor problems, so the two defences are in tension and you must choose knowingly.

The final, under-appreciated one is alignment luck. Where the hot loop happens to land relative to cache lines and branch-predictor structures can change results measurably, and adding an unrelated function elsewhere in the binary can shift it. This is a genuine source of irreproducible differences between builds, and it is why a difference of a few percent between two versions is usually noise rather than signal.

A checklist worth running before believing any CPU microbenchmark
[ ] Does elapsed time scale with trip count?         -> if not, the work was deleted
[ ] Is the result consumed opaquely?                 -> if not, it can be deleted
[ ] Is the input distribution production-like?       -> if not, predictor and cache are unrealistic
[ ] Is the working set production-sized?             -> if not, miss costs are understated
[ ] Did it run long enough to leave boost?           -> if not, the frequency is unsustainable
[ ] Is the operation >> timer cost, or batched?      -> if not, you measured the clock
[ ] Repeated across rebuilds / alignments?           -> if not, a few percent is noise
[ ] Machine otherwise idle and pinned?               -> if not, neighbours are in the number

Key points

  • A compiler may remove work whose result is unobservable; verify by scaling the trip count and checking the time scales too.
  • Benchmarks run fully warmed — trained predictor, resident cache, mapped pages — which is the best case and rarely the production case.
  • Input distribution matters as much as the code: benchmarking sorted or uniform data measures a predictor at its best.
  • Short runs measure a boost frequency the machine cannot sustain; run to steady state and report that.
  • Timer cost and code alignment both perturb results, so differences of a few percent between builds are usually noise.

Progressive depth

Overview

A microbenchmark measures the code you wrote under conditions you did not intend: warm caches, a trained predictor, a boosted clock, and possibly no work at all if the optimiser removed it.

Practical

Consume the result opaquely, check that time scales with trip count, use realistic inputs and working-set sizes, run long enough to leave boost, and repeat across rebuilds before believing a small difference.

Advanced

Timer amortisation and cache realism pull in opposite directions, so choose deliberately. Alignment luck sets a noise floor of a few percent that no amount of repetition within one build will reveal — only rebuilding does.

Internals

Frequency during a short run is governed by boost algorithms responding to power and thermal headroom accumulated while idle, so the first measurement after a pause is systematically unrepresentative. Alignment effects arise from how the loop maps onto front-end structures whose indexing is undocumented, which is why they appear as unexplained build-to-build variance.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    Source → compiler: unobservable computation is legally removable, and optimisers do remove it.
  2. 2
    Loop → predictor: after a few hundred iterations every branch in the loop is predicted correctly.
  3. 3
    Loop → caches: the benchmark working set becomes fully resident, and stays resident because nothing else runs.
  4. 4
    Short run → boost: the core runs above its sustainable frequency for the duration of the measurement.
  5. 5
    Timer → measurement: reading the clock costs cycles that are attributed to the operation unless amortised across a batch.
What people conclude from this — wrongly
  • • Believing an impossibly good result instead of suspecting the optimiser.
  • • Treating a warmed-up per-operation cost as the cost the same operation has inside a real program.
  • • Reading a few percent difference between builds as a real improvement.
  • • Concluding a change is good from a microbenchmark alone, without an end-to-end check.

Consequences, controls and cost

What it causes
  • • Published per-operation costs are frequently the fully warmed best case, understating production cost several-fold.
  • • Optimisations validated only by microbenchmark regularly fail to move any production metric.
  • • Two builds of identical source can differ by a few percent purely through code alignment.
  • • A benchmark that was accidentally deleted by the optimiser reports a result so good it is rarely questioned.
What you can do
  • • Make results observable and independently verify by checking that time scales with trip count.
  • • Use production-representative input distributions and working-set sizes, not convenient ones.
  • • Run long enough to leave boost and reach thermal steady state; report the sustained number.
  • • Batch operations to amortise timer cost, while remaining aware this warms the caches further.
  • • Repeat across rebuilds and treat small differences as noise until they survive that.
  • • Confirm any microbenchmark conclusion against an end-to-end measurement before acting on it.
How to see it
  • • Trip-count scaling as the primary sanity check that the work happened at all.
  • • Instructions retired for the benchmark region, compared against the expected instruction count.
  • • Frequency over the run, to confirm the measurement was not taken entirely in boost.
  • • Repeat runs across rebuilds to establish the noise floor before comparing versions.
  • • An end-to-end measurement of the enclosing workload as the final arbiter.
What it costs
  • • Defences against dead-code elimination add overhead that is itself included in the measurement.
  • • Batching to amortise timer cost warms caches and predictors further, which is the opposite of realism.
  • • Production-representative benchmarks are much harder to build and much slower to run than convenient ones.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • MICROARCH-SPECIFICBoost behaviour, timer instruction cost and alignment sensitivity differ by vendor and generation, so the size of each of these effects must be established on the target machine.
  • PLATFORM-SPECIFICTimer resolution and cost, frequency-governor policy and the ability to pin threads all depend on the operating system and on whether the machine is virtualized.

Misconceptions

Claim
“The loop ran ten million times, so the work definitely happened.”
Reality
Not if the results are unused. Compilers remove unobservable computation, and the tell is a per-operation cost that is impossibly small or does not scale with trip count.
Claim
“A warm-up phase makes the benchmark realistic.”
Reality
Warm-up makes it *repeatable*, which is different. It moves the measurement toward the best case, while production is usually somewhere much colder.
Claim
“This build is 3% faster, so the change worked.”
Reality
Three percent is inside the noise produced by code alignment alone. It needs to survive repeated rebuilds before it is a result.

Where the rest of this lives

Programming Languages & Runtime Internals
JIT warm-up and deoptimisation

On a managed runtime the first thousands of iterations run interpreted or in a lower tier, so a benchmark that does not account for tiering measures the compiler rather than the code.

← Your Code Is Data TooThe Clock Is a Variable →