Latency and Bandwidth Are Different Resources
A workload can saturate memory bandwidth while barely being affected by latency, or be crippled by latency while using a fraction of available bandwidth. Conflating the two sends people to the wrong fix — and "the memory is slow" is almost never a complete diagnosis.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Two independent limits
Consider two loops over the same amount of data. The first sums a large contiguous array. The second walks a linked list scattered across that same footprint. Both touch a similar number of bytes. The first can have many loads outstanding simultaneously because the addresses are all computable in advance, so it pushes the memory system toward its bandwidth ceiling. The second cannot issue the next load until the current one returns, so it has exactly one request in flight and sits at the mercy of latency.
The result is that the streaming loop may be running at ninety percent of peak memory bandwidth while the pointer-chasing loop uses a tiny fraction of it — and yet the pointer-chasing loop takes longer. Adding memory channels helps the first and does essentially nothing for the second. Reducing the dependency chain helps the second and does nothing for the first.
This is why "the memory is slow" is not a diagnosis. The actionable question is whether you are bandwidth-bound or latency-bound, and the two have their own lessons: When the Memory Bus Is the Bottleneck and When You Cannot Ask the Next Question Yet.
| Latency-bound | Bandwidth-bound | |
|---|---|---|
| Limiting quantity | Time for one dependent access | Bytes per second delivered |
| Typical shape | Pointer chasing, dependent loads, tree walks | Streaming, scanning, large copies |
| Outstanding requests | Few — often one | Many |
| Memory bandwidth used | Low | Near peak |
| Adding cores | Helps if work is independent | Does not help; bandwidth is already saturated |
| What actually helps | Shorten dependency chains, prefetch, restructure to arrays | Move fewer bytes: compression, smaller types, better layout |
The dependency chain is the difference
The mechanism separating the two cases is memory-level parallelism — how many misses the core can have outstanding at once. Modern cores can track a number of concurrent misses, and this is what turns a sequence of high-latency accesses into a high-bandwidth stream: if ten misses are in flight together, the effective cost per access is a tenth of the latency.
A dependency chain destroys this. If the address of the next load is the *result* of the current load, then by construction only one can be in flight. No amount of hardware parallelism helps, because the hardware does not know the next address yet. This is the whole story of Pointer Chasing: The Address You Do Not Have Yet.
The comparison below is the canonical demonstration. Both loops read the same number of elements from the same array; only the address dependency differs.
1// idx[] has been shuffled, so each load's address2// comes from the value the previous load returned3i = 04for (n = 0; n < N; n++) {5 i = idx[i] // must wait for this load6 sum += data[i] // before this one can even be addressed7}8 9// outstanding misses: 110// bandwidth used: a fraction of peak11// limited by: DRAM latency x N1for (i = 0; i < N; i++) {2 sum += data[i] // address is i, computable immediately3}4 5// the core can issue many loads before the first returns,6// and the prefetcher runs ahead of them7 8// outstanding misses: many9// bandwidth used: near peak10// limited by: bytes/second the memory system deliversIdentical byte counts, identical instruction counts, wildly different runtimes. The difference is not how much memory is touched but whether the addresses are known early enough to overlap the accesses. This is the clearest demonstration in the domain that data *movement* and data *dependency* are separate costs.
Diagnosing which one you have
The diagnosis is usually quick. Measure achieved memory bandwidth during the hot phase and compare it against what the machine can sustain for a pure streaming benchmark. If you are near that ceiling, you are bandwidth-bound and the fix is to move fewer bytes. If you are far below it while stalling on memory, you are latency-bound and the fix is to get more requests in flight.
A second, cruder test: add cores. A bandwidth-bound workload stops scaling once the memory system saturates, and can even regress as contention rises. A latency-bound workload with independent work per core scales roughly linearly, because each core independently has its one outstanding request.
This diagnosis is the hardware-level counterpart of the reasoning in Busy Is Not the Same as Working, and it feeds directly into the When the Memory Bus Is the Bottleneck and When You Cannot Ask the Next Question Yet lessons.
- Near-peak bandwidth plus high stall time → bandwidth-bound. Move fewer bytes.
- Low bandwidth plus high stall time → latency-bound. Get more accesses in flight.
- Low bandwidth plus low stall time → not a memory problem at all; look at Busy Is Not the Same as Working.
- Scaling stops when cores are added → bandwidth or another shared resource has saturated.
Key points
- Latency is time-per-access; bandwidth is bytes-per-second in aggregate. They are separate resources with separate limits.
- Memory-level parallelism converts high-latency accesses into a high-bandwidth stream — but only when addresses are known in advance.
- A dependency chain forces one outstanding access at a time, which is why pointer chasing is latency-bound no matter how fast the memory is.
- Bandwidth-bound workloads stop scaling with more cores; latency-bound workloads with independent work usually keep scaling.
- The two conditions have opposite fixes, so distinguishing them before acting is the whole value of the diagnosis.
Progressive depth
Overview
Latency is how long one memory access takes. Bandwidth is how much data the memory system can move per second. "Slow memory" could mean either, and they need different fixes.
Practical
Measure achieved bandwidth against the machine's streaming ceiling. Near the ceiling means bandwidth-bound: move fewer bytes with smaller types, better packing or compression. Far below it while stalling means latency-bound: shorten dependency chains, use arrays instead of pointer structures, or prefetch explicitly.
Advanced
The bridge between them is memory-level parallelism. A core can sustain a limited number of outstanding misses; with k misses in flight the effective per-access cost approaches latency/k. Streaming code saturates that capability and becomes bandwidth-limited; dependent code cannot use it at all. This is why unrolling a pointer chase does not help — unrolling exposes instruction-level parallelism, not memory-level parallelism, because the addresses still are not known.
Internals
At the DRAM level the two limits have different physical origins. Latency is dominated by the activate-read-precharge protocol described in How DRAM Is Organised, plus queueing delay in the memory controller. Bandwidth is set by bus width and clock, degraded by refresh, row conflicts and read-write turnaround. Because the controller reorders requests to maximise row hits, a workload with many independent outstanding requests also gets *better* row-buffer behaviour than one issuing them serially, so the two effects compound in favour of parallel access.
Where the Data Is
Change an input and watch which number moves — and which one refuses to.
The exact ratios vary by machine and the absolute times vary far more, which is why none are shown. What is stable enough to build intuition on is the shape: each level is several times the one above, and the gap between the last cache level and memory is the one that decides most program performance.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Core → load-store unit: a load issues and, if it misses, occupies one of a limited number of miss-tracking entries.
- 2Miss entries → memory controller: several independent misses can be outstanding simultaneously, overlapping their latencies.
- 3Dependency → parallelism: if the next address depends on a returning value, no second miss can be issued, so only one entry is ever occupied.
- 4Controller → DRAM: many concurrent requests let the controller batch row hits and use multiple banks, approaching bandwidth limits.
- 5Bandwidth ceiling → stall: once the bus saturates, additional requests simply queue, and adding cores stops helping.
- • "We are memory-bound, so we need faster RAM." Faster RAM raises bandwidth and barely moves latency; if you are latency-bound this buys almost nothing.
- • "Bandwidth usage is low, so memory is not the problem." Low bandwidth with high stall time is the *signature* of a latency problem.
- • "Adding threads will fix it." It will if you are latency-bound with independent work, and will not if bandwidth is already saturated.
Consequences, controls and cost
- • Two loops touching identical amounts of data can differ several-fold in runtime purely because of address dependencies.
- • Bandwidth-bound code stops scaling with core count, sometimes regressing as contention grows.
- • Optimisations that reduce instruction count can leave a memory-bound loop completely unchanged.
- • Diagnose first: compare achieved bandwidth against the machine's streaming ceiling before choosing a remedy.
- • For bandwidth limits, move fewer bytes — smaller types, tighter packing, avoiding fields you do not read (see [[aos-vs-soa]]).
- • For latency limits, break dependency chains so multiple accesses can be outstanding, or convert pointer structures into arrays.
- • Use software prefetching only after the simpler restructurings, and only with measurement — it is easy to make things worse.
- • Read achieved memory bandwidth from performance counters during the hot phase and compare against a pure streaming benchmark on the same machine.
- • Count outstanding misses or measure stall cycles attributed to memory; low bandwidth with high stalls indicates latency-bound behaviour.
- • Run a core-count scaling sweep: flat scaling implicates a saturated shared resource, near-linear scaling suggests per-core latency limits.
- • Reducing bytes moved often means compression or narrower types, spending CPU cycles to save bandwidth — a good trade only when bandwidth is genuinely the limit.
- • Breaking dependency chains typically means restructuring data layout, which costs abstraction and code clarity.
Scope
§224 — what these claims are specific to.
- GENERALThe latency-versus-bandwidth distinction holds on every machine with a cache hierarchy. The number of outstanding misses a core can track is MICROARCH-SPECIFIC and differs substantially between designs.