Instruction-Level Parallelism
A single thread, on a single core, with no threading library anywhere in sight, routinely has a dozen operations in flight at once. That is ILP — parallelism the hardware extracts from your sequential code without being asked, and the first thing to understand before reaching for threads.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Parallelism you already have
Instruction-level parallelism is the overlap the hardware finds inside one instruction stream. No API exposes it, no library enables it and no OS scheduler is involved. The core simply looks at the operations in its window, finds ones that do not depend on each other, and runs them at the same time on different units.
This is worth stating plainly because it changes the order in which you should try things. A loop achieving well under one instruction per cycle on a machine capable of several is leaving a large multiple on the table *within the core it is already running on*. Adding threads to such a loop parallelises the inefficiency; fixing the dependency structure raises the per-thread rate and then threading multiplies the improved number.
ILP is also the reason the naive cost model fails. If a core can issue several operations per cycle and each has multi-cycle latency, then "how many instructions" and "how many cycles" are related by a factor the code shape controls — the ratio IPC: Instructions Per Cycle measures.
| Limit | What it means | How it shows up | Where to read |
|---|---|---|---|
| True dependencies | Each operation needs the previous result | Low IPC, low cache misses, idle ports | Dependency Graphs: The Real Shape of Your Code |
| Memory latency | Operations wait on loads that miss | Low IPC with high miss counters | Hits, Misses and What a Miss Actually Costs |
| Branch mispredictions | Speculative work discarded, window refilled | Low IPC with high misprediction counters | Misprediction: What a Wrong Guess Costs |
| Port contention | Too many operations of one kind competing | Moderate IPC that will not rise with unrolling | Superscalar Execution |
| Window capacity | Reorder buffer full, front end stalled | Stalls despite idle execution units | The Reorder Buffer and Precise State |
| Front-end supply | Decoder cannot deliver instructions fast enough | Frontend-bound counters, large hot code | Your Code Is Data Too |
ILP is not thread-level parallelism
These are two entirely different mechanisms and conflating them produces bad decisions. ILP is extracted by hardware from *one* instruction stream, needs no coordination, has no synchronisation cost and cannot cause a data race. Thread-level parallelism uses *several* instruction streams, is created and scheduled by software, and brings the full weight of shared-memory correctness with it.
They also scale against different limits. ILP is capped by the dependency structure of your code and the width of one core — you cannot unlock more of it by buying a machine with more cores. Thread-level parallelism is capped by how much of the work is independent at a coarse grain, and by the coherence and synchronisation traffic that appears once several cores touch the same data (Cache Coherence: Why Shared Memory Works At All, False Sharing: Independent Data, Shared Line).
A useful discipline: get ILP right first, because it is free of correctness risk, and only then decide whether the remaining work justifies threads. The full four-way distinction — instruction, data, thread and core level — is laid out in Four Kinds of Parallelism.
Diagnosing a thread that is not using its core
The diagnostic is a two-number comparison. Measure the achieved instructions per cycle, then compare against the core's maximum issue width. A thread sustaining a small fraction of what the core can issue is not compute-bound in any useful sense — it is waiting, and the counters say what for.
The decision tree is short. High cache-miss counters point at memory and send you to the memory hierarchy. High misprediction counters point at control flow and send you to Branch Prediction: Guessing Well Enough to Matter. Low counts on both, with low IPC, points at a dependency chain — the machine has work but the work is serialised. Each branch has a different fix, and guessing between them is how optimisation effort gets wasted.
The counters and their interpretation are treated properly in The CPU Counts Itself and in the Observability & Performance domain; what matters here is the framing. "The CPU is at 100%" says a core is occupied, not that it is doing useful work — a core stalled on memory reports as busy while achieving very little.
BEFORE (array of structs, one field touched) instructions 1.00e9 cycles 2.40e9 IPC 0.42 <- core can issue several per cycle L1-dcache-misses 8.10e7 <- most accesses miss branch-misses 1.20e6 <- not the problem AFTER (struct of arrays, same algorithm) instructions 1.00e9 cycles 0.61e9 IPC 1.64 L1-dcache-misses 9.40e6 branch-misses 1.20e6 same work, 3.9x fewer cycles, zero threads added
Key points
- ILP is parallelism the hardware extracts from a single thread; it requires nothing from the programmer and carries no correctness risk.
- A low IPC on a wide core means the thread is not using the core it already has — fix that before adding threads.
- ILP and thread-level parallelism are different mechanisms with different limits; more cores do not give a thread more ILP.
- The limits on ILP are dependencies, memory latency, mispredictions, port contention, window capacity and front-end supply.
- "CPU at 100%" means occupied, not productive: a core stalled on memory looks identical to one doing useful work.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Front end → window: instructions are decoded and placed into the scheduling window as fast as the decoder can supply them.
- 2Window → dependence tracking: each operation records outstanding operands; those with none are eligible immediately.
- 3Eligible set → issue: several operations are selected per cycle, limited by issue width and by which ports each needs.
- 4Ports → execution: independent operations proceed concurrently on separate units, which is the parallelism itself.
- 5Stall source → IPC: whichever of dependency, miss, misprediction or window capacity binds first determines the achieved rate.
- • "Single-threaded means one operation at a time." A single thread routinely has many operations in flight.
- • "My CPU is at 100%, so I am compute-bound." Stalled cores report as busy; check IPC and stall counters before concluding.
- • "More cores will fix this loop." Cores do not increase the ILP available to one thread.
- • "IPC is low, so the CPU is slow." Low IPC is a symptom with several possible causes, and the counters distinguish them.
- • "ILP is an optimisation I can turn on." It is always on; what you control is whether your code contains independent work for it to find.
Consequences, controls and cost
- • Well-structured single-threaded code can sustain several instructions per cycle on a wide core; poorly structured code sustains a fraction of one.
- • Threading a chain-limited loop multiplies an inefficiency rather than removing it.
- • The same source compiled for a narrow in-order core behaves very differently, because the hardware extracts far less on its own.
- • Occupancy metrics from the OS cannot distinguish a productive core from a stalled one.
- • Improvements in layout or chain structure often deliver larger factors than adding a second thread, and with none of the synchronisation risk.
- • Measure IPC against the core's issue width before deciding what kind of problem you have.
- • Remove the binding limit indicated by the counters — dependencies, misses or mispredictions — rather than optimising generally.
- • Restructure data so that independent work exists to be found ([[data-oriented-design]], [[aos-vs-soa]]).
- • Only after per-thread efficiency is reasonable, consider thread-level parallelism for the remaining work.
- • On in-order targets, expect the compiler's static scheduling to matter far more, since the hardware will not compensate.
- • Compute IPC from instruction and cycle counters and compare it against the documented issue width of the target core.
- • Read cache-miss and branch-miss counters alongside IPC; the combination identifies which limit is binding.
- • Use top-down style stall categories where available to split frontend-bound, backend-bound, bad speculation and retiring.
- • Re-measure after a single targeted change — layout, chain structure, branch removal — to confirm the limit moved.
- • Repeat on each target microarchitecture; the binding limit can differ between a wide desktop core and a narrow embedded one.
- • Restructuring for ILP costs readability and couples the code to assumptions about the machine.
- • Optimising per-thread efficiency first delays the parallel speedup, though it usually makes the eventual parallel version better.
- • Techniques that expose ILP — unrolling, multiple accumulators, wider working sets — can increase register pressure and code size.
- • Time spent reading counters is time not spent on algorithmic improvement, which often has a larger ceiling.
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICAchievable IPC depends on issue width, port mix and window size. A wide desktop core and a small embedded core differ by a large factor on identical code, and neither number is portable.
- PLATFORM-SPECIFICCounter names and availability differ by vendor, OS and virtualisation. Inside many virtual machines and containers, hardware counters are restricted or unavailable entirely; see The Hardware That Makes Virtual Machines Possible.
Misconceptions
Apply it
Where the rest of this lives
The decision of whether to parallelise across threads, and what it costs in synchronisation and correctness risk, is a concurrency question. This lesson only establishes that per-thread efficiency should be settled first.