10 lessons

Hardware Performance Analysis

Reading the machine: performance counters, CPI, telling compute-bound from memory-bound, and every way a microbenchmark will lie to you about frequency, caches and dead code.

SourceCompilerInstructionsFront EndExecutionRegistersCachesMemoryI/OBehavior
The CPU Counts Itself
▶ lab

Every modern CPU carries a small unit whose only job is to tally what the rest of the chip did: cycles, instructions retired, misses at each cache level, mispredicted branches, stalled cycles. It is the only direct evidence you will ever get about the hardware — and it is sampled, approximate, and named differently on every chip.

Q · How does the CPU report what it was actually doing, and how much of that report can I believe?
CPI and IPC: The Number Everyone Misreads

Cycles per instruction, and its reciprocal instructions per cycle, describe how smoothly work is flowing through the machine. Neither is a measure of performance. A change that halves CPI while tripling instruction count made the program slower, and a vectorized loop that runs twice as fast often shows a worse CPI than the scalar loop it replaced.

Q · What does cycles-per-instruction actually tell me about my program, and why is a better CPI sometimes a worse program?
Busy Is Not the Same as Working

A core showing 100% utilisation may be executing a dense stream of arithmetic, or it may be stalled almost the entire time waiting for data that has not arrived. The operating system reports both as "busy". They are different problems with disjoint fixes, and only the counters can tell them apart.

Q · The core is pinned at 100%. Is it doing work, or is it waiting — and how would I know the difference?
Misses That Overlap Are Nearly Free

A cache miss costs a great deal if the core has nothing else to do, and almost nothing if it does. Modern cores keep several misses outstanding at once, so ten independent misses can cost barely more than one — while ten dependent misses cost ten times as much. This is why miss counts alone never predict runtime.

Q · Why do two loops with the same number of cache misses take completely different amounts of time?
Your Code Is Data Too

Instructions are fetched from memory through their own cache, and that cache is small. A hot loop that fits runs at full speed; a sprawling call graph with aggressive inlining can spend a large fraction of its cycles waiting for instructions to arrive — a stall that data-focused profiling is structurally unable to see.

Q · Why does a program with excellent data locality still stall, and why can inlining make it slower?
Every Way a CPU Microbenchmark Lies

A microbenchmark measures what it measures, which is frequently not what you meant. The compiler deletes work whose result is unused, the caches and predictors are warm in ways production never is, the clock speed moves underneath you, and the timer itself costs more than the operation. Each of these has produced published results that were simply wrong.

Q · Why does my microbenchmark say this code is fast when production says otherwise?
The Clock Is a Variable

The number printed on the box is a nominal figure, not an operating one. Real clock frequency moves continuously with load, thermal headroom, power budget, how many cores are active, and even which instructions are executing — wide vector code frequently runs at a lower clock than scalar code on the same chip.

Q · Why do identical runs of the same code take different amounts of time on the same machine?
The First Ten Seconds Lie

Silicon has a temperature limit, and the only lever the chip has to stay under it is to slow down. A workload that starts on a cool chip runs at one speed and settles at a lower one, which is why burst performance and sustained performance are different numbers and why short benchmarks systematically flatter the machine.

Q · Why does a long-running workload get slower over time even though nothing about the code changed?
Performance Per Watt

Energy, not time, is the constraint that actually binds on phones, on laptops and across datacentres. The configuration that finishes soonest is frequently not the one that uses least energy, and the relationship between the two is non-linear enough that "run slower to save power" is often exactly wrong.

Q · When is the fastest configuration not the right one, and how does energy behave differently from time?
Throughput Improved, Latency Did Not

Nearly every technique modern CPUs use — pipelining, superscalar issue, out-of-order execution, speculation — increases the number of operations completed per unit time without reducing, and sometimes while increasing, the time any single operation takes. This is why decades of architectural progress leave a dependent chain almost exactly as slow as it was.

Q · Why has all this architectural progress made parallel work so much faster and dependent work barely faster at all?