The question this answers
Adding workers stopped helping but every core still looks busy — am I out of CPU, or out of memory bandwidth?
A streaming aggregation over a 40 GB columnar dataset: read each 8-byte value, add it to a running total, discard it. Almost no arithmetic per byte, and no reuse.
No program-level shared state at all — each worker owns a disjoint slice and a private accumulator. The shared thing is the hardware: the memory controllers, the channels to DRAM and the shared last-level cache, none of which appear anywhere in the source.
Every value is read exactly once and added to exactly one worker's accumulator; the sum of accumulators equals the sum over the dataset, at every worker count.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The curve that flattens with the CPU still busy
This job has no locks, no shared writes and no barriers until the end. By every rule of thumb it should scale linearly, and up to about four workers it nearly does. Then it stops — 8 workers is barely better than 4, and 16 is no better than 8 — and the profiler shows every core at high utilization the whole time. That combination is the signature: busy but not productive.
A core waiting on a cache miss is counted as busy by every operating-system utilization metric, because it is not idle and it has not yielded. What has actually happened is that the memory subsystem is delivering all the bytes per second it can, and each additional worker just takes a share of a fixed pie. Per-worker throughput falls in almost exact proportion to worker count, which is the cleanest confirmation available: 8 workers each doing an eighth of the work of one worker is a saturated shared resource, not a scheduling problem.
This is the single most misdiagnosed scaling wall, because the reflex is to look for locks. There are none. Thread dumps show every thread running. Lock-wait metrics are zero. The bottleneck is in a place that no concurrency tool instruments, and the only way to see it directly is to look at instructions retired per cycle, or at bytes moved per second against the machine's known ceiling.
Arithmetic intensity predicts the wall before you hit it
You can tell which side of this line a loop falls on before running it, by counting operations per byte moved. total += a[i] over doubles is one add per eight bytes: 0.125 operations per byte, which is about as bandwidth-bound as code gets. A dense matrix multiply performs roughly n operations per element loaded because each element is reused n times once resident, which is why it is compute-bound and why it is the workload accelerators are designed around.
The read-out below does the arithmetic for a concrete machine shape. The conclusion — that four or five cores are enough to saturate a socket for a streaming loop — is the useful part, because it explains the shape of the curve above and it tells you that the fix is not more parallelism.
It also explains an unintuitive interaction with SIMD: One Instruction, Many Elements: vectorizing a bandwidth-bound loop makes each core demand bytes *faster*, so the bus saturates at fewer cores. Vectorization is still correct — you reach the same ceiling with less hardware, which is cheaper — but the speedup you observe will be far below the instruction-count improvement, and that is not a bug in the vectorization.
- Count operations per byte before parallelizing. Under about 1 op/byte, expect a bandwidth wall at a handful of cores.
- Vectorizing a bandwidth-bound loop moves the wall to *fewer* cores, because each core consumes its share faster.
- Hyperthread siblings share the same path to memory, so they add almost nothing to a bandwidth-bound workload while looking like extra cores.
machine (assumed): ~100 GB/s sustained to DRAM, cores at ~3 GHz loop A: total += a[i] (float64 streaming sum) bytes per element .......... 8 operations per element ..... 1 arithmetic intensity ....... 0.125 ops/byte <- BANDWIDTH BOUND one core wants ............. 3e9 * 8 B = 24 GB/s cores to saturate 100 GB/s . ~4 => scaling stops near 4 workers, whatever the core count loop B: C = A x B (dense matmul, blocked) bytes per element .......... 8, but reused ~n times from cache operations per byte ........ grows with block size <- COMPUTE BOUND => scales with cores until the cores run out the diagnostic pair, per worker count: CPU utilization HIGH + IPC FALLING -> memory bound CPU utilization HIGH + IPC FLAT -> compute bound (good) CPU utilization LOW -> synchronization or I/O, not memory and the confirmation: per-worker throughput. if 8 workers each do 1/8 of what 1 worker did, a shared resource is full.
What to do when the bus is the bottleneck
Every effective fix moves in one direction: do more work per byte, or move fewer bytes. Adding workers, adding threads, or replacing a lock with an atomic does none of that and changes nothing. This is the most valuable thing to internalize here, because it redirects effort from concurrency techniques — which are the reflex — to data techniques, which are the only ones that work.
Fusing passes is usually the largest available win and the least glamorous: three sequential loops over a 40 GB array move 120 GB; one loop doing all three operations moves 40 GB. Same results, one third of the traffic, and the speedup is close to 3x on a saturated machine. Compression is the same trade in a different currency — spend CPU cycles (which you have) to move fewer bytes (which you do not) — and it frequently wins outright on a bandwidth-bound job even though it looks like extra work.
Layout is the other lever. Reading one field from an array-of-structs pulls entire structs through the cache, so you pay for every field you did not want; a columnar layout moves only the column you asked for. This is exactly why analytics systems are columnar, and it is a bandwidth argument rather than a storage one. Parallelism Can Destroy Locality covers the neighbouring problem of workers destroying each other's cache warmth.
| Move | Effect on bytes moved | Cost | Typical win |
|---|---|---|---|
| Add more workers | None — same bytes, more waiters | More cores, more scheduling cost | Zero. This is the reflex and it does nothing |
| Fuse multiple passes into one | Divides traffic by the number of passes fused | Loop body gets more complex; less modular code | Large and reliable |
| Compress data in memory | Divides traffic by the compression ratio | CPU cycles to decode — which you have spare | Large when the ratio is good |
| Columnar / struct-of-arrays layout | Moves only the fields actually read | Pervasive change to the data type and its consumers | Large for wide records read narrowly |
| Block or tile for cache reuse | Same bytes from DRAM, far more reuse per byte | Non-trivial algorithm rewrite | Large for anything with reuse (matmul, joins) |
| Filter earlier / read less | Directly proportional | Predicate pushdown plumbing | Largest of all when applicable |
| Replace a lock with an atomic | None | Time and correctness risk | Zero — the lock was never the problem |
Key points
- Cores are multiplied; the path to memory is shared. Streaming workloads saturate the second one long before the first.
- The signature is high CPU utilization with falling instructions-per-cycle, and per-worker throughput falling roughly as 1/N.
- A stalled core counts as busy, which is why every standard utilization metric hides this bottleneck.
- Arithmetic intensity — operations per byte moved — predicts which side of the wall a loop is on before you write the parallel version.
- Every real fix reduces bytes moved or increases work per byte. Adding workers or removing locks does neither.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Each core issues loads; misses go to the shared last-level cache and then to the memory controllers.
- • The controllers and DRAM channels deliver a bounded number of bytes per second, shared across all cores in the socket.
- • One core cannot usually saturate that ceiling, so early workers scale nearly linearly and the workload looks compute-bound.
- • At some worker count the aggregate demand meets the ceiling; from then on each additional worker receives a smaller share and total throughput is constant.
- • Cores are now stalled waiting on memory. They are not idle, so utilization stays high while useful work per cycle falls.
- • There is no program-level interleaving here: workers touch disjoint slices with private accumulators, and every schedule produces the same result. That is exactly what makes it confusing — the correctness reasoning finds nothing wrong.
- • The hardware-level interleaving that matters: W0 issues a load; W1 issues a load; W2 issues a load; the controller queues them and serves them in some order — each worker's latency now includes the others' requests.
- • The one program-level schedule that can appear anyway: the final combine, where eight partial sums are added. Two workers reading and writing a shared total lose an update; give each worker its own slot and add them once at the end.
- • A related and much worse case: workers whose accumulators sit on the same cache line. Nothing is logically shared, but the line ping-pongs between cores and throughput collapses — see False Sharing: Different Variables, Same Cache Line, which is bandwidth's evil twin and has a much sharper signature.
- • Disjoint slices guarantee correctness at any worker count, with no synchronization — this lesson is about performance, not safety.
- • The hardware guarantees coherence: a load always returns the most recent value, whatever the traffic cost of providing it.
- • Nothing guarantees a scaling benefit past the saturation point, and nothing in the language, runtime or OS warns you that you have passed it.
- • Vectorization guarantees fewer instructions; it does NOT guarantee proportionally less time when the loop is bandwidth-bound.
- • Utilization metrics guarantee only that a core was not idle. They say nothing about whether it retired any instructions.
- • The memory controllers and DRAM channels are the contended resource, and they are contended by every process on the machine — including co-tenants you cannot see.
- • The shared last-level cache is contended too: eight workers streaming different slices evict each other's lines, so effective cache size per worker shrinks as workers grow.
- • Hyperthread siblings share the core's path to memory, so they contribute little to a bandwidth-bound workload while consuming a "core" in every pool-sizing calculation.
- • On a multi-socket machine, remote memory access adds interconnect contention on top of local bandwidth contention — see NUMA: Not All Memory Costs the Same.
- • Flat scaling past a low worker count, with no lock, no barrier and no I/O to blame.
- • A misdiagnosis that spends weeks on lock-free data structures for a job that never had lock contention.
- • A benchmark that scales beautifully because its input fits in cache, followed by a production workload that does not and does not scale at all.
- • Vectorization delivering 4x fewer instructions and 1.1x less time, reported as "SIMD did not work".
- • Noisy-neighbour variance on shared hosts: the same job scales differently on different days because someone else is using the bus.
- • Pool sizing based on logical core count, adding hyperthread siblings that cannot help this workload and do add scheduling cost.
- • Recognizing the pattern early: an ops-per-byte estimate takes minutes and can save a parallelization effort that was never going to pay.
- • Sizing worker pools honestly for streaming jobs — four workers that saturate the bus cost less than sixteen that do the same throughput.
- • Justifying data-layout and compression work, which is otherwise a hard sell because it looks like it adds work.
- • When it becomes the explanation for every flat curve. Synchronization and I/O produce flat curves too, and they show low CPU rather than high.
- • When it is used to avoid parallelizing compute-bound work that would have scaled fine — check intensity, do not assume.
- • When teams optimize bytes for a job whose real constraint is a serial phase or a barrier.
- • Instructions retired per cycle, per worker count. Falling IPC with rising workers is the definitive memory signature.
- • Bytes per second moved to and from DRAM, compared with the machine's known sustained ceiling — the direct measurement, if the counters are available.
- • Per-worker throughput: if N workers each achieve 1/N of a single worker, a shared resource is saturated.
- • Cache miss rate at the last level, which distinguishes "working set does not fit" from "genuinely streaming".
- • Run the same job with half the workers. Equal throughput at half the cores is proof, and it needs no hardware counters at all.
- • The fixes are data-layout and algorithm changes, not concurrency changes — they touch types, serialization formats and every consumer of the data.
- • Fusing passes trades modularity for traffic: one loop doing four things is faster and harder to read, test and reuse.
- • In-memory compression adds an encoding, a decoder in the hot path, and a new failure mode when the ratio is bad for some inputs.
- • Measuring this well requires hardware performance counters, which are often unavailable in containers and shared cloud environments — so the practical fallback is the half-the-workers experiment.
- • Read less: predicate pushdown, projection, better indexes, or a summary table. Not moving a byte beats moving it efficiently.
- • Scale out instead of up: two machines have two memory subsystems, so a bandwidth-bound job scales across hosts even when it will not scale across cores.
- • Change the storage format so the layout matches the access pattern, turning a wide-record scan into a narrow-column scan.
- • Accept four workers and spend the remaining cores on other work — a saturated bus is a capacity fact, and co-locating a compute-bound job on the same machine uses what the streaming job cannot.
Why is 8 cores only 4.5×?
| workers | ideal | Amdahl only | realistic | limited by |
|---|---|---|---|---|
| 1 | 1.0× | 1.00× | 1.00× | none |
| 2 | 2.0× | 1.90× | 1.85× | serial |
| 4 | 4.0× | 3.48× | 3.19× | serial |
| 6 | 6.0× | 4.80× | 4.17× | serial |
| 8 | 8.0× | 5.93× | 4.17× | bandwidth |
| 10 | 10.0× | 6.90× | 4.17× | bandwidth |
| 12 | 12.0× | 7.74× | 4.17× | bandwidth |
| 14 | 14.0× | 8.48× | 4.17× | bandwidth |
| 16 | 16.0× | 9.14× | 4.17× | bandwidth |
Two counters, no lock, one cache line
struct Counters {
long a; // byte 0..7
long b; // byte 8..15 <- same 64-byte line as a
}; // sizeof == 16CPU parallelism simulator
Amdahl’s term, a synchronisation term, an oversubscription term and a bandwidth ceiling, each one a knob you can switch off. Real curves have more causes than four and are rarely this smooth. There is no ideal core count to read off this chart.
What people believe, and what is true
CPU is at 90%, so we are CPU-bound.
A core stalled on a cache miss is counted as busy. High utilization with low instructions-per-cycle means the core is waiting for memory, not computing.
It stopped scaling, so there must be lock contention.
Lock contention shows up as *low* CPU utilization and measurable blocked time. A bandwidth wall shows high utilization and zero blocked time.
This machine has 64 cores, so we can run 64 workers on anything.
For a streaming workload the bus fills at a handful. The other workers add scheduling cost and cache pollution in exchange for no throughput.
SIMD did not help, so the compiler failed to vectorize.
On a bandwidth-bound loop, fewer instructions cannot mean less time — the bytes still have to arrive. Check the intensity before blaming the vectorizer.