Main Memory & DRAM
Past the last-level cache: how DRAM is organized, why latency and bandwidth are different resources, and how to tell a bandwidth-bound workload from a latency-bound one.
When every cache misses, the request leaves the CPU entirely. It goes to a memory controller that queues it, reorders it against other pending requests, and drives a DRAM device that is nothing like the flat byte array your program believes in.
DRAM is not a flat array. It is a grid of rows and columns across banks, and reading it means activating a whole row into a buffer first. Whether your next access hits that open row or forces another activation is a several-fold cost difference nothing in your code mentions.
A workload can saturate memory bandwidth while barely being affected by latency, or be crippled by latency while using a fraction of available bandwidth. Conflating the two sends people to the wrong fix — and "the memory is slow" is almost never a complete diagnosis.
Streaming code that touches each byte once cannot be helped by caches, cannot be helped by more cores, and cannot be helped by faster arithmetic. It is limited by how fast bytes arrive, and the only real lever is moving fewer of them.
Some loops use almost no memory bandwidth and are still dominated by memory. Each access must complete before the next address is even known, so the hardware's ability to overlap misses is worth nothing, and the loop runs at one DRAM round trip per step.