Pipeline Hazards: The Three Ways Overlap Fails
Pipelining assumes the next instruction can always start. Three situations break that assumption — a needed value is not ready, the next address is not known, or two instructions want the same hardware — and every hardware performance problem is a variation on one of them.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Data hazards: the value is not ready
The classic case: one instruction produces a value the next instruction consumes. In a naive pipeline the consumer reaches the stage where it needs the operand before the producer has written it back, so it must wait.
Most of these are eliminated in practice by forwarding — routing the result directly from where it was produced to where it is needed, without waiting for write back (Forwarding and Stalls: Paying for Dependencies). Forwarding handles the arithmetic-to-arithmetic case completely.
The case forwarding cannot fully fix is load-use: an instruction that needs a value the immediately preceding load is fetching. The value simply does not exist yet, because the cache has not returned it. On a hit that is a short stall; on a miss it is hundreds of cycles, and the chained version of this is Pointer Chasing: The Address You Do Not Have Yet, the hardest access pattern in the domain.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
|---|---|---|---|---|---|---|---|
| I1 LOAD r1, [r9] | I | I | E | M | W | ||
| I2 ADD r2, r1, r4 | I | I | E | M | W |
Control hazards: the next address is not known
When the pipeline reaches a conditional branch, the condition has usually not been evaluated yet. The fetch stage needs an address *now*, several stages ahead of where the branch will resolve. Without any mechanism, the pipeline would stall until resolution, which on a deep pipeline is ruinous.
The mechanism is prediction: guess the direction and target, fetch from there, and continue speculatively. When the guess is right — which on well-behaved code it overwhelmingly is — there is no cost at all. When it is wrong, the speculative work is discarded and fetch restarts, costing roughly the pipeline depth in wasted cycles.
That asymmetry is the important part. Branch cost is not "branches are slow"; it is "*unpredictable* branches are slow". A loop condition taken a million times and not taken once is essentially free. A branch driven by random data is expensive every time, which is why Branch Prediction: Guessing Well Enough to Matter and Misprediction: What a Wrong Guess Costs are separate lessons.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
|---|---|---|---|---|---|---|---|---|
| I1 BEQ r1, r2, target | I | I | E | M | W | |||
| I2 (wrong path, discarded) | I | I | ||||||
| I3 (wrong path, discarded) | I | |||||||
| I4 (correct target) | I | I | E | M | W |
Structural hazards: the hardware is busy
Two instructions need the same resource in the same cycle and the machine only has one of them. A single memory port shared between instruction fetch and data access is the textbook example; in practice, separate L1 caches remove that one, and modern structural hazards are subtler — a limited number of load units, one divider, a shared vector unit between SMT threads.
Structural hazards are the least discussed of the three and the most likely to be misdiagnosed, because the symptom is "low IPC with no misses and no mispredictions". If the loop performs three loads per iteration on a core with two load units, no amount of cache tuning or branch restructuring will help. The fix is a different instruction mix: fewer loads, different data layout (Array of Structs, or Struct of Arrays?), or fewer iterations.
This is also where SMT: Two Contexts, One Core shows its cost. Two hardware threads on one core share the functional units, so a structural hazard that was invisible with one thread can dominate with two — and this is precisely why SMT helps some workloads and hurts others.
| Hazard | What it means | Counter signature | What fixes it |
|---|---|---|---|
| Data | Operand not ready yet | Low IPC; high cache misses if load-related | Break dependency chains; improve locality |
| Control | Next address unknown | High branch-miss rate | Make branches predictable, or remove them |
| Structural | Required unit is busy | Low IPC with few misses of any kind | Change instruction mix; fewer loads; check SMT |
Key points
- Three hazards break pipeline overlap: data (value not ready), control (address unknown), structural (unit busy).
- Forwarding removes most data hazards, but cannot remove load-use — the data genuinely does not exist yet.
- Branch cost is a function of predictability, not of branch count.
- Structural hazards present as low IPC with clean miss and mispredict counters, and are the most misdiagnosed.
- Each hazard has a distinct fix; identifying which one dominates is the whole diagnostic value.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Decode → dependency check: the machine determines whether an instruction's operands are available or still in flight.
- 2Unavailable operand → stall or forward: a result already computed can be forwarded; one not yet produced forces a bubble.
- 3Branch → speculative fetch: the front end continues from a predicted address before the condition resolves.
- 4Misprediction → squash and refill: speculative instructions are discarded and fetch restarts, costing roughly pipeline depth.
- 5Unit contention → issue stall: an instruction ready to execute waits because every unit capable of executing it is occupied.
- • "Low IPC means the CPU is memory-bound" — it can equally be a dependency chain or a structural limit, and the counters distinguish them.
- • "Branches are slow" — predictable branches are close to free; unpredictability is the cost.
- • "Removing instructions will help" — only if the removed instructions were the ones causing bubbles.
Consequences, controls and cost
- • Loops with tight dependency chains run at a fraction of the machine's issue width despite perfect cache behaviour.
- • Data-dependent branches on unsorted input can cost more than the work the branch guards.
- • Load-heavy loops plateau at the core's load-unit count regardless of arithmetic simplification.
- • Identify which hazard dominates before changing anything — the three fixes are unrelated and applying the wrong one wastes the effort.
- • For data hazards: multiple accumulators, unrolling, and better locality so loads hit cache.
- • For control hazards: make the branch predictable (sort the data, hoist the condition) or eliminate it.
- • For structural hazards: reduce the count of the contended operation, usually loads, via layout changes.
- • Read branch-miss rate, cache-miss rate and IPC together — the combination identifies the hazard, no single counter does.
- • A top-down breakdown, where the vendor provides one, attributes stalls to front end, bad speculation or back end directly.
- • Test structurally: change one thing (sort the input, add an accumulator, reduce loads) and see which moves the number.
- • Hazard-driven optimisation is microarchitecture-specific and can regress on a different CPU generation.
- • The transformations involved — unrolling, sorting, branch removal — all cost readability and sometimes memory.
- • Time spent here is wasted if the loop is actually memory-bandwidth-limited, which is why the diagnosis must come first.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDThe five-stage pipeline shown resolves branches early and has one execution unit. Real cores predict earlier, resolve later and have many units, which changes the magnitudes but not the taxonomy.
- MICROARCH-SPECIFICUnit counts, forwarding paths and mispredict penalties differ per design; a structural hazard on one core may not exist on another with more load units.