Why Reading the Source Cannot Tell You the Cost
Two adjacent lines of source imply neither two instructions nor two steps in time. Source code specifies *what result is required*, and it is an excellent tool for reasoning about correctness — but it deliberately says nothing about instruction count, ordering or cost, which is exactly why measurement exists.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
What source order actually specifies
Source order specifies a *dependency and observability contract*, not a schedule. It says: whatever the machine does, the result must be as if these statements had run in this order, judged by the effects the language calls observable. That is a constraint on outcomes, and a deliberately loose one on mechanism, which is precisely what allows a compiler to vectorize a loop and a CPU to execute two hundred instructions concurrently.
So source reasoning is authoritative for exactly one class of question: what result will this produce, under the language's rules? Correctness, invariants, types, and the behaviour of a single thread all follow from the source and should be reasoned about there. This is not a weak claim — it is the entire reason high-level languages work, and it is why "just read the code" is good advice for a correctness bug.
It is authoritative for approximately nothing about cost. Instruction count, execution order, memory traffic, cycles, parallelism and energy are all properties of the compiled program running on a specific machine with a specific data set, and none of them is recoverable from the text. The failure is not that source reasoning is bad; it is applying a correctness tool to a cost question.
| Question | Answerable from source? | Why |
|---|---|---|
| What value does this produce? | Yes — this is what source is for | The language defines the result exactly |
| Is this thread-safe under the language's model? | Yes, with the memory model in hand | Ordering guarantees are a language-level contract |
| How many instructions will run? | No | Inlining, unrolling, vectorization and elision all intervene |
| In what order will operations execute? | No | Compiler and CPU both reorder (The Compiler Reordered It Before the CPU Did) |
| How long will this line take? | No | Depends on residency, prediction and contention — none of it in the text |
| Which line is the bottleneck? | No — measure it | Attribution requires a profiler; intuition is famously wrong here |
The same line, three different costs
Consider sum += arr[i] inside a loop. If arr[i] is already in L1 because the previous iteration brought its line in, the load is nearly free and the add is one of several the core issues that cycle. If the index jumped and the line is not resident, the same source line stalls for the time it takes to reach DRAM — two orders of magnitude more (Hits, Misses and What a Miss Actually Costs). If the loop vectorizes, the line does not correspond to one operation at all but to one lane of a wide instruction processing many elements at once.
Nothing distinguishes these three cases in the text. The only difference is the *access pattern and the data size*, which are runtime properties. This is why the domain insists on measurement: not because reasoning is useless, but because the determining variables are not present in the artifact being reasoned about.
The inverse error is just as common and more expensive: assuming a line that *looks* costly is costly. A division looks expensive and is, but it may be off the critical path and fully overlapped. A function call looks expensive and may have been inlined to nothing. A "simple" pointer dereference looks free and may be the most expensive operation in the function (Pointer Chasing: The Address You Do Not Have Yet). Cost intuition trained on source appearance is close to random.
1total += node->value // one dereference2node = node->next // one dereference3 4// Each step depends on the previous load.5// Nothing can be prefetched, nothing overlaps.6// The core is idle for most of this loop.1total += a[i] * b[i] + c[i] // three loads, a multiply, two adds2 3// Contiguous, prefetchable, independent across iterations.4// The compiler may vectorize it; the CPU overlaps many5// iterations at once. More arithmetic, less waiting.Counting operations in the source ranks these backwards. The second line does strictly more arithmetic and is dramatically faster, because its accesses are predictable and independent while the first is a serialised chain of dependent memory accesses. Operation count is not cost; data movement and dependency structure are.
What to do instead
Use source reasoning for what it is good at and switch tools for cost. In practice: reason about correctness in the source, form a *hypothesis* about cost from the source, and then measure to confirm or kill it. The hypothesis matters — measuring without one produces numbers nobody can interpret — but it must be held loosely, because the base rate of correct performance intuition is low even among experts.
The measurement ladder runs roughly: wall-clock timing to know whether a problem exists, a profiler to know where, performance counters to know why (The CPU Counts Itself, CPI and IPC: The Number Everyone Misreads), and the emitted assembly to know what actually ran. Most questions are settled at the second rung; the interesting ones need the third.
This is the practical form of the Engineer Atlas thesis. Abstractions are worth using — nobody should hand-schedule instructions — but an abstraction you cannot see through is one you cannot debug. The goal of this entire domain is not to make you distrust high-level code; it is to give you a model of what is underneath, so that when the abstraction leaks, the leak is legible rather than magic.
- Correctness — reason in the source. It is the authority, and this is what it is for.
- Cost hypothesis — form one from the source, then treat it as a guess to be tested.
- Where — profiler. Intuition about which line is hot is unreliable even for experts.
- Why — performance counters: misses, mispredictions, stalls, IPC.
- What actually ran — the emitted assembly, which ends most arguments immediately.
Key points
- Source order specifies a contract about results, not a schedule of operations — that looseness is what permits optimisation.
- Source reasoning is authoritative for correctness and near-useless for cost, and confusing the two is the recurring error.
- The same line can cost a fraction of a cycle or hundreds, depending on residency and dependencies that do not appear in the text.
- Cost intuition trained on how expensive code *looks* ranks operations close to randomly.
- The remedy is a ladder: hypothesis from source, profiler for where, counters for why, assembly for what actually ran.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Source → semantics: the language fixes required results and leaves mechanism unconstrained beyond observable effects.
- 2Semantics → compiler: transformations reshape the instruction stream so that statement-to-instruction correspondence is lost (The Compiler Reordered It Before the CPU Did).
- 3Instructions → CPU: fetch is in order, execution is by readiness, and many instructions are in flight simultaneously (Out-of-Order Execution).
- 4Execution → memory: the dominant term is often whether an operand was resident, which depends on runtime data rather than on code (The Memory Hierarchy).
- 5Memory → observed cost: the resulting wall-clock time is a property of code, data, machine and neighbours together — not of the source alone.
- • "This line has a loop in it so it is the bottleneck" — loop presence says nothing about trip count, residency or vectorization.
- • "Fewer lines of code means fewer instructions" — abstraction level and instruction count are almost unrelated after inlining.
- • "The profiler must be wrong, this line is trivial" — a trivial-looking line is exactly where a cache miss surfaces.
- • "We reasoned carefully, so we do not need to measure" — careful reasoning about the wrong variables is still wrong.
Consequences, controls and cost
- • Optimisation effort routinely lands on code that was never the bottleneck, because it looked expensive.
- • Reviews reject readable code for imagined performance reasons that measurement would not support.
- • Two systems running the same source can have very different performance profiles because their data differs.
- • Engineers who have never measured tend to hold strong and inconsistent beliefs about what is fast.
- • Measure before optimising, and measure the metric users actually experience rather than a microbenchmark.
- • Write down the cost hypothesis before profiling, so a wrong guess is visibly wrong and recalibrates you.
- • Prefer clear code by default; buy performance only where measurement shows it is needed.
- • Learn to read the emitted assembly for hot functions — it converts arguments into facts quickly.
- • Keep a representative data set: performance conclusions from unrepresentative data are worse than none.
- • Time the whole operation first to establish whether a problem exists at all before profiling anything.
- • Profile to attribute cost to functions and lines, treating line attribution as approximate.
- • Read counters to distinguish stalled cycles from executing cycles, which separates memory problems from compute ones.
- • Inspect the emitted assembly for the hot function to confirm what the compiler actually produced.
- • Measurement infrastructure takes effort to build and keep representative as the system changes.
- • Profiling perturbs the thing being measured, and sampling attribution is approximate near inlined boundaries.
- • Chasing measured hot spots can produce locally fast, globally unreadable code if applied without judgement.
- • Learning to read assembly and counters is a real time investment that pays back only for those who work near the metal.
Scope
§224 — what these claims are specific to.
- GENERALApplies to every optimising toolchain and every out-of-order processor. The gap is narrower for interpreted languages executing a fixed bytecode loop, but present there too because the interpreter itself runs on the same hardware.
- SIMPLIFIEDThe comparison uses two illustrative loops to make the ranking vivid; actual relative costs depend on data size, residency, compiler version and machine.
Misconceptions
Apply it
Where the rest of this lives
Interpretation, bytecode, JIT tiers and deoptimisation add another layer between text and instructions, one that can change the emitted code while the program is running.