Pipelining: Throughput Without Making Anything Faster
A pipelined CPU does not execute any single instruction more quickly than an unpipelined one. It overlaps them, so instructions complete more often. Understanding that pipelining buys throughput and not latency explains most of what modern CPUs do and why they do it.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The laundry argument, made precise
Divide the work of an instruction into stages that use different hardware: fetch uses the instruction cache, decode uses the decoders, execute uses a functional unit, write back uses the register file. In an unpipelined design, each of those sits idle while the others work, and one instruction completes every five cycles.
Pipelining puts a different instruction in each stage simultaneously. After the pipeline fills, one instruction completes every cycle — a five-fold improvement in throughput from exactly the same hardware, with no stage made faster.
The individual instruction is not helped at all. It still takes five cycles from fetch to write back, and in practice slightly more, because pipeline registers between stages add a small overhead. This is the trade: latency stays flat or worsens marginally, throughput multiplies. Every other property of modern CPUs — deep pipelines, branch prediction, out-of-order execution — follows from pushing on that trade.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|
| I1 LOAD r1, [r9] | I | I | E | M | W | ||||
| I2 ADD r2, r3, r4 | I | I | E | M | W | ||||
| I3 SUB r5, r6, r7 | I | I | E | M | W | ||||
| I4 AND r8, r9, r10 | I | I | E | M | W | ||||
| I5 XOR r11, r12, r13 | I | I | E | M | W |
Deeper pipelines, and what they cost
If five stages give five-way overlap, why not fifty? Because two costs grow with depth. First, the pipeline must be *refilled* whenever control flow goes somewhere unpredicted, and the refill cost is proportional to depth — this is what makes Misprediction: What a Wrong Guess Costs expensive, and it is why prediction accuracy matters more on deep pipelines.
Second, deeper pipelines mean shorter stages, which permits a higher clock but does less per cycle. Chasing clock frequency by lengthening the pipeline was tried extensively in the early 2000s and largely abandoned: the mispredict penalty and the power cost outgrew the frequency gain. Modern designs sit at moderate depths and buy performance from width and out-of-order execution instead.
This is the concrete reason The Clock: Why GHz Is Not Performance insists that gigahertz is not performance. Two cores at the same frequency with different pipeline depths, widths and prediction accuracy will differ substantially on the same code, and the deeper one is not automatically the faster.
| Property | Shallow pipeline | Deep pipeline |
|---|---|---|
| Work per stage | More | Less |
| Achievable clock frequency | Lower | Higher |
| Instruction latency in cycles | Fewer stages to traverse | More stages to traverse |
| Misprediction penalty | Small — few cycles to refill | Large — proportional to depth |
| Sensitivity to branch quality | Modest | High; prediction accuracy dominates |
| Power at a given performance | Generally lower | Generally higher |
What breaks the overlap
Pipelining assumes the next instruction can enter the pipeline immediately. Three things break that assumption, and they are the subject of Pipeline Hazards: The Three Ways Overlap Fails: an instruction needing a result not yet produced (data), an instruction whose address is not yet known (control), and two instructions needing the same hardware in the same cycle (structural).
Each hazard costs a bubble — a cycle in which a stage produces nothing. Bubbles are the visible mechanism of essentially every hardware performance problem: a cache miss is a very long bubble, a mispredicted branch is a series of them, a dependency chain is a bubble every few instructions.
The mental model to leave with is a factory line, not a queue of work. The question that predicts performance is not "how many instructions" but "how often does the line stop, and for how long". That reframing is what makes counters like IPC: Instructions Per Cycle readable and what Busy Is Not the Same as Working operationalises.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
|---|---|---|---|---|---|---|---|---|
| I1 LOAD r1, [r9] | I | I | E | M | W | |||
| I2 ADD r2, r1, r4 | I | I | E | M | W | |||
| I3 SUB r5, r6, r7 | I | I | E | M | W |
Key points
- Pipelining improves throughput, not the latency of any individual instruction.
- In steady state a filled pipeline completes roughly one instruction per cycle from hardware that would otherwise complete one per five.
- Deeper pipelines allow higher clocks but make mispredictions proportionally more expensive.
- Bubbles — stages producing nothing — are the visible mechanism behind almost every hardware performance problem.
- The predictive question is "how often does the line stop", not "how many instructions are there".
Pipeline Simulator
Change an input and watch which number moves — and which one refuses to.
A teaching model. Real cores have far more stages, several instructions per stage, and execute out of order — but the hazards below are real and behave the same way.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
|---|---|---|---|---|---|---|---|---|
| ADD R1,R2,R3 | F | D | E | M | W | |||
| SUB R4,R5,R6 | F | D | E | M | W | |||
| AND R7,R8,R9 | F | D | E | M | W | |||
| OR R10,R11,R12 | F | D | E | M | W |
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Stage partition → parallel occupancy: the instruction path is divided so each stage uses distinct hardware and can host a different instruction.
- 2Pipeline fill → steady state: after depth-many cycles every stage is busy and completions occur once per cycle.
- 3Dependency → bubble: an instruction whose operand is not ready waits, and the stage behind it produces nothing that cycle.
- 4Branch → redirect: an unpredicted control transfer discards the partially-executed instructions behind it and refills from the correct address.
- 5Refill cost → depth: the number of wasted cycles after a redirect scales with how deep the pipeline is.
- • "Pipelining makes instructions faster" — it makes them complete more often; each one still takes just as long.
- • "A deeper pipeline is a better CPU" — it enables a higher clock and raises the cost of every misprediction.
- • "My loop has few instructions so it must be fast" — a short loop full of dependent operations can run at a fraction of peak.
Consequences, controls and cost
- • Instruction throughput can approach one per cycle even though each instruction takes several cycles individually.
- • Code with long dependency chains runs far below the machine's peak rate with no cache misses to blame.
- • Branch-heavy code with unpredictable outcomes underperforms in proportion to pipeline depth.
- • Provide independent work so the pipeline has something to overlap: multiple accumulators, unrolled loops, interleaved computations.
- • Make branches predictable — sorted data, hoisted conditions, or removing the branch entirely ([[branchless-code]]).
- • Reduce loads on the critical path, since a cache miss is the longest bubble available.
- • Nothing about the pipeline itself is under programmer control; the leverage is entirely in what you feed it.
- • Compute IPC for the hot loop: values well below the machine's issue width indicate bubbles, and the counters tell you which kind.
- • Use a top-down breakdown where available to attribute stalls to front end, back end, bad speculation or retiring.
- • Compare a loop against a deliberately independent version of itself; a large gap confirms dependency-induced bubbles.
- • Creating independent work usually means unrolling, which costs code size and instruction-cache pressure.
- • Restructuring for predictability can obscure the algorithm and make maintenance harder.
- • Optimisations targeting bubbles are microarchitecture-sensitive and may not transfer between CPU generations.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDThe five-stage model is a teaching device. Real pipelines are deeper, wider, and include rename, schedule and retire stages absent here; the hazard behaviour it illustrates is nonetheless real.
- MICROARCH-SPECIFICDepth, width and refill cost differ per design and are often unpublished. Treat depth-dependent costs like the mispredict penalty as qualitative, not as a number that transfers.