7 lessons

Out-of-Order & Parallelism

Modern CPUs are throughput machines: many instructions in flight, executed as their inputs become ready, retired in program order. Renaming, superscalar issue, ILP and what IPC actually tells you.

SourceCompilerInstructionsFront EndExecutionRegistersCachesMemoryI/OBehavior
Out-of-Order Execution
▶ lab

You wrote A, B, C. If B is waiting on a cache miss, the machine will run C first — and then hand you a result indistinguishable from having run them in order. This is the lesson where "the CPU executes my code line by line" stops being a useful model.

Q · If the CPU is free to execute my instructions in a different order than I wrote them, why does my program still produce the answer I expect?
Dependency Graphs: The Real Shape of Your Code

Program order is a line. What the machine actually obeys is a graph — and the longest path through that graph, not the number of nodes in it, is what sets the floor on how fast a loop can run.

Q · Two loops execute the same number of instructions and one is three times slower — what is the machine actually constrained by?
Instruction-Level Parallelism

A single thread, on a single core, with no threading library anywhere in sight, routinely has a dozen operations in flight at once. That is ILP — parallelism the hardware extracts from your sequential code without being asked, and the first thing to understand before reaching for threads.

Q · How much parallelism is my single-threaded code already getting for free, and what stops it from getting more?
Superscalar Execution

A pipelined core finishes one instruction per cycle at best. A superscalar core has several execution units and finishes several — provided your instructions need different units and do not depend on each other. Port contention is why the theoretical peak is theoretical.

Q · What actually determines how many instructions my core can complete in one cycle?
Register Renaming

The ISA gives you a handful of register names. Reusing one creates a dependency that has nothing to do with your data — a naming collision, not a real ordering requirement. Renaming maps those names onto a much larger physical file and the false dependency disappears.

Q · If the ISA only defines a small number of registers, how does the CPU keep hundreds of operations in flight without them constantly colliding?
The Reorder Buffer and Precise State

Execution finishes in whatever order the data allows. Something has to put the results back in order before anyone can see them — and that same something is what lets a page fault, an interrupt or a mispredicted branch unwind cleanly instead of corrupting your program.

Q · If operations complete out of order, what makes an exception land at exactly the right instruction?
IPC: Instructions Per Cycle

The ratio that connects "how much work" to "how long it took". It is the most useful single number for diagnosing a CPU-bound loop — and one of the easiest to misuse, because a change that raises IPC can leave the program slower.

Q · What does instructions-per-cycle actually tell me, and when does improving it make the program slower?