Superscalar Execution
A pipelined core finishes one instruction per cycle at best. A superscalar core has several execution units and finishes several — provided your instructions need different units and do not depend on each other. Port contention is why the theoretical peak is theoretical.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Width is a mix, not a number
It is tempting to describe a core as "four-wide" and treat that as four of anything per cycle. Real cores distribute their execution units across ports with different capabilities. There may be several ports that can do simple integer arithmetic but only one that can do integer division, two that can issue loads but only one that can issue a store address, and a separate set for vector operations.
So the achievable rate for a specific loop depends on the *shape* of its instruction mix relative to the port layout. A loop of independent integer adds may saturate the ALU ports and reach a high rate. A loop of independent divisions may reach one per several cycles, because there is one divider and it is not fully pipelined. Both loops contain nothing but independent work; only the port they need differs.
This is the third distinct limit on throughput, after dependencies (Dependency Graphs: The Real Shape of Your Code) and the front end. It is also the one that unrolling cannot fix — unrolling supplies more independent work, and port contention is precisely the situation where more independent work has nowhere to go.
| 1 | 2 | 3 | 4 | |
|---|---|---|---|---|
| add r1, r2, r3 [ALU port 0] | i | e | ||
| add r4, r5, r6 [ALU port 1] | i | e | ||
| load r7, [r8] [load port] | i | e | ||
| load r9, [r10] [load port] | i | e | ||
| div r11, r12 [divider] | i | e | e |
Front end, back end, and which one is starving
A superscalar back end can only issue what the front end delivers. Fetching, decoding and — on some designs — translating instructions into internal operations all have their own width limits, and a loop whose hot code is large or badly laid out can starve a perfectly capable back end.
That gives two failure modes that look similar from a distance and require opposite fixes. Backend-bound means operations are waiting on execution resources or data: the answer is fewer dependencies, better locality or a different instruction mix. Frontend-bound means the scheduler is idle because instructions are not arriving: the answer is smaller hot code, better branch density or fewer instruction-cache misses (Your Code Is Data Too).
Modern counters expose this split directly, which is why the top-down methodology starts by asking which side is limiting before asking anything else. Guessing at this level is unusually expensive because the two fixes actively work against each other — aggressive unrolling helps the back end and hurts the front end.
| Backend-bound | Frontend-bound | |
|---|---|---|
| What is idle | The window is full; ports or data are the constraint | The window is empty; instructions are not arriving |
| Typical cause | Dependency chains, cache misses, port contention | Instruction-cache misses, large hot loop, decode limits |
| Effect of unrolling | Often helps — more independent work | Often hurts — more code to fetch |
| Effect of inlining | Can help by removing call overhead | Can hurt by growing the hot footprint |
| What to read | Dependency Graphs: The Real Shape of Your Code, Hits, Misses and What a Miss Actually Costs | Your Code Is Data Too, Branch Prediction: Guessing Well Enough to Matter |
What this means when you tune
The practical consequence is that "add more independent work" is a fix with a ceiling, and the ceiling is set by the port layout. Once a loop saturates the port its critical operation needs, the only remaining moves are to use a different port — a cheaper operation, a different formulation — or to do less work per element.
The vector ports are the interesting case here. A scalar loop saturating an ALU port has one obvious escape: issue the same arithmetic through the vector units instead, processing several elements per instruction. That converts a port-contention problem into a throughput win without needing any more independent work, which is one reason SIMD: One Instruction, Many Elements is such a large lever on numeric loops.
This is also the point at which per-machine tuning stops generalising. Port counts and capabilities differ across vendors and generations; a mix tuned to saturate one layout can be unbalanced on another. Measure on the target, and treat any specific port assignment you read about as MICROARCH-SPECIFIC by default.
- Dependency-limited — supply independent work: unroll, use several accumulators.
- Port-limited — change the instruction mix: cheaper operations, or move the work to vector units.
- Front-end-limited — shrink the hot code: less unrolling, less inlining, better layout.
- Memory-limited — none of the above helps; fix locality first (Spatial Locality).
Key points
- Superscalar means several instructions issued per cycle, but only to ports that can execute them.
- Port contention is a distinct limit from dependency chains, and unrolling does not fix it.
- A core is not "N-wide" for all instruction types; the mix relative to the port layout decides the rate.
- Backend-bound and frontend-bound look similar and need opposite fixes — measure which one binds.
- Vector units are a separate set of ports, which is why vectorising can break through a scalar port limit.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Front end → window: instructions are decoded and supplied at the front end's own width limit.
- 2Window → readiness: operations whose operands are available become candidates for issue.
- 3Candidates → port arbitration: each operation needs a specific port; the scheduler picks a compatible subset for this cycle.
- 4Port → execution unit: chosen operations execute concurrently on separate units.
- 5Non-pipelined unit → occupancy: some units accept a new operation only every several cycles, serialising that instruction type regardless of independence.
- • "The core is four-wide, so I should get four instructions per cycle." Only if the mix matches the ports.
- • "Unrolling more will keep helping." It helps dependency limits, not port limits, and eventually hurts the front end.
- • "Independent work always issues in parallel." Independent work competing for one port issues serially.
- • "Low IPC means memory." It can equally mean port contention or a starved front end; the counters distinguish them.
- • "This port assignment I read about applies to my CPU." It applies to the microarchitecture it was documented for.
Consequences, controls and cost
- • A loop can plateau at a rate well below the core's headline width purely from instruction mix.
- • Unrolling improves dependency-limited loops and can degrade front-end-limited ones.
- • Mixing operation types in a loop body sometimes raises throughput by spreading work across more ports.
- • Divisions and some transcendental operations serialise far more than their instruction count suggests.
- • Tuning that balances a specific port layout does not transfer to a different core.
- • Determine whether the loop is front-end or back-end limited before changing anything.
- • If port-limited on scalar arithmetic, consider vectorising to move work to different units ([[vectorization]]).
- • Replace expensive single-port operations — division by multiplication by a reciprocal where precision allows, for example.
- • Balance the instruction mix so that not every operation competes for the same unit.
- • Stop unrolling once it stops helping; past that point it is only costing instruction-cache footprint.
- • Split stalls into frontend-bound and backend-bound categories using top-down counters before tuning.
- • Compare achieved IPC against the core's documented issue width for the specific instruction mix in the loop.
- • Vary the instruction mix experimentally — swap an operation for a cheaper one on a different port — and watch whether throughput moves.
- • Unroll progressively and record where improvement stops; the plateau identifies a non-dependency limit.
- • Re-run on each target microarchitecture, since the binding port differs between designs.
- • Balancing an instruction mix for a specific port layout is machine-specific tuning with a short shelf life.
- • Replacing division with reciprocal multiplication changes floating-point results.
- • Unrolling to expose parallelism costs code size and register pressure.
- • Vectorising to escape a scalar port limit adds complexity and portability concerns ([[auto-vectorization]]).
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICIssue width, port count, which operations each port supports and which units are pipelined are all specific to a core design and change between generations. Even within one vendor, performance and efficiency cores in the same package differ.
- SIMPLIFIEDThe pipeline trace models issue and execute only, and shows instructions rather than the internal micro-operations a real decoder produces. Register renaming, dispatch queues and retirement are omitted.