What a Second Core Actually Adds
Eight cores is not one core that goes eight times faster. It is eight execution engines, each with private caches, sharing one last-level cache and one memory system through an interconnect — and that shared half is where multicore performance is usually won or lost.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Private per core, shared across cores
A modern multicore chip is best read as two layers. The private layer is a complete CPU repeated N times: its own register file, its own execution units, its own branch predictor, its own L1 data and instruction caches, and on most designs its own L2. Two threads running in these layers genuinely do not interfere — this is where linear scaling comes from.
The shared layer is everything past that: a last-level cache shared by some or all cores, an interconnect carrying coherence traffic and cache-line transfers, one or more memory controllers, and the DRAM channels behind them. Every core reaching past its private caches competes here. A workload whose working set fits in private cache scales beautifully; the same workload at four times the data can flatten completely, having changed nothing but its size.
This is why "how many cores" is a much weaker predictor of parallel throughput than "how much of the working set stays private". It is also why two benchmarks of the same algorithm can disagree entirely — one sized to fit L2, the other not.
Which resource runs out first
Scaling behaviour follows directly from which shared resource saturates. If threads mostly hit private cache, adding cores adds throughput. If they stream large arrays, they saturate memory bandwidth and the curve flattens — more cores then add heat, not work. If they share written data, they saturate the coherence interconnect, and adding cores can make throughput go *down*, because each additional core adds invalidation traffic without adding useful work.
That last case is the one that surprises people, so it is worth stating plainly: parallel speedup is not monotonic. There are real workloads whose best configuration is four threads on a sixteen-core machine, and the reason is always in the shared layer.
| Workload shape | Resource that saturates | Scaling curve |
|---|---|---|
| Working set fits private L1/L2, little sharing | None — cores are independent | Near-linear; the good case |
| Large read-only streaming over arrays | Memory bandwidth and LLC capacity | Flattens once bandwidth is saturated |
| Threads write to shared lines | Coherence interconnect | Can invert — more cores, less throughput |
| Threads sharing one LLC with hostile access patterns | LLC capacity, via mutual eviction | Degrades as thread count rises |
| Mostly blocked on I/O | Nothing on-chip | Limited by the I/O, not the cores |
The shared half is where the cost lives
Reaching a private L1 is the cheapest thing a core can do short of a register. Reaching the shared LLC costs meaningfully more. Fetching a line that another core currently owns in a modified state is more expensive still, because it requires a transfer between caches rather than a lookup. Reaching DRAM is the most expensive of all.
Those four rungs are the entire performance story of multicore programming, and the ordering is stable across every machine even though the ratios are not. The lesson to carry forward is directional: keep data private, keep it small, and do not write to lines other cores are reading. Cache Coherence: Why Shared Memory Works At All explains the third of those, and it is the one with the least intuitive cost.
Key points
- A core duplicates registers, execution units and private caches; it does not duplicate the LLC, interconnect, memory controller or DRAM.
- Parallel scaling is decided by which shared resource saturates first, not by the core count.
- A workload that fits in private cache scales near-linearly; the same workload at four times the size may not scale at all.
- Writing to shared cache lines can make throughput fall as cores are added, because coherence traffic grows without useful work.
- The cost ordering private L1 → private L2 → shared LLC → remote-owned line → DRAM is stable everywhere; the ratios are not.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Thread → core: the OS schedules a software thread onto a logical CPU, which has its own registers and private caches.
- 2Core → private L1/L2: most accesses are served here with no interconnect involvement and no other core aware of them.
- 3Core → interconnect: a miss in private cache becomes a request onto the shared fabric, where it queues behind every other core's traffic.
- 4Interconnect → LLC or peer cache: the line is supplied from the shared cache, or transferred from whichever core currently owns it.
- 5LLC → memory controller → DRAM: a full miss goes to memory, consuming bandwidth shared by every core on the chip.
- • "Eight cores means eight times the throughput" — true only when nothing in the shared layer saturates, which is a narrow case.
- • "It scaled linearly in the benchmark, so it will scale in production" — the benchmark probably fit in cache and production does not.
- • "Throughput dropped when we added threads, so the scheduler is broken" — far more often it is coherence traffic on shared lines.
- • "More cores is always the better machine" — a chip with fewer cores and more cache per core can win on cache-sensitive work.
Consequences, controls and cost
- • Speedup curves flatten at a thread count that depends on the working-set size, not on the core count.
- • The same code scales on a laptop and fails to scale on a server part with more cores but a different LLC arrangement.
- • Benchmarks sized to fit private cache report scaling that production, with larger data, never reproduces.
- • Adding threads to a coherence-bound workload actively reduces throughput.
- • Size the per-thread working set to stay in private cache where the algorithm allows — this is the single largest lever.
- • Partition data so that each thread owns its slice, rather than threads sharing structures and coordinating.
- • Measure scaling as a curve across thread counts rather than assuming; the peak is often not the maximum thread count.
- • When throughput falls as threads are added, stop adding threads and look at the coherence traffic before anything else.
- • Run the workload at 1, 2, 4, 8 … threads and plot throughput. The shape of that curve is the diagnosis.
- • Watch LLC miss rate as threads are added; a rise means the threads are evicting each other from the shared cache.
- • Watch memory bandwidth utilisation; approaching the platform maximum explains a flat curve completely.
- • Compare a run pinned to cores on one cluster or socket against an unpinned run — a large gap points at the shared topology.
- • Partitioning data per thread removes sharing but duplicates memory and can multiply the total footprint.
- • Keeping working sets small often means smaller batches, which costs per-item overhead.
- • Tuning to one machine's cache topology produces code that may perform worse on the next machine.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDA flat set of cores over one shared LLC. Chiplet and mesh designs partition the LLC into slices with non-uniform access cost, and hybrid CPUs mix core types with different cache sizes entirely.
- MICROARCH-SPECIFICWhether L2 is private or shared by a pair of cores, and how many cores share an LLC, varies by vendor and generation; some server parts share an LLC only within a cluster.
Misconceptions
Where the rest of this lives
This lesson explains the *hardware* ceiling on parallel speedup. The algorithmic ceiling — the serial fraction that no amount of hardware removes — is a concurrency topic, and the two ceilings compound.