What a Cache Actually Is
A cache is not a faster memory. It is a small tagged store holding copies of recently used lines, managed entirely by hardware, betting that your program will ask for the same or nearby data again. When the bet pays it is invisible; when it fails it is also invisible, which is the problem.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
A copy, a tag, and a bet
Three ideas make a cache. First, it holds copies — the authoritative value still lives further out, which is precisely why keeping multiple copies coherent across cores becomes a problem (Cache Coherence: Why Shared Memory Works At All). Second, every entry carries a tag recording which memory address the copy came from, because a small store cannot have a slot per address and must be able to answer "is this the line you asked for?". Third, it is a bet on locality: keeping this line is only worth the space if you are going to ask for it, or its neighbours, again.
That bet is the reason caches work rather than just adding indirection. Real programs loop over data, revisit the same structures, and walk contiguous ranges. A cache converts that regularity into speed automatically. A program with genuinely random access over a large range defeats it completely, and no amount of cache will help — which is the honest version of "caches are not magic".
The hardware does all of this without being asked. There is no instruction in ordinary code that says "cache this". Some ISAs offer hints — non-temporal stores that bypass the cache, or explicit prefetch instructions — but these are advisory, situational, and easy to make things worse with.
Hardware-managed means you steer, not drive
The distinction that matters practically: you do not decide what is cached, but you decide what is *asked for*, in what order, and how densely it is packed. Every real cache optimisation is one of those three, applied indirectly.
This is a different relationship than you have with an application-level cache, where you call get and put and choose the eviction policy. Conflating the two leads to wrong instincts — people ask how to "pin" a value in L1, which is not a thing ordinary code can do, when the productive question was how to make the hot data smaller or the traversal denser.
It is also a different thing from the OS page cache, which is RAM the kernel uses to hold file contents and which *is* somewhat controllable through system calls. Keeping those two straight is a §224 requirement in this domain and has its own lesson, CPU Cache Is Not the Page Cache, because conflating them produces confident nonsense about both.
| CPU cache | OS page cache | Application cache | |
|---|---|---|---|
| Managed by | Hardware, autonomously | Kernel | Your code |
| Holds | Lines of physical memory | Pages of file data in RAM | Whatever objects you put in it |
| Direct control | None in ordinary code; hints only | Advisory syscalls and flags | Full — you choose keys, size and policy |
| Eviction policy | Undocumented recency approximation | Kernel policy, tunable in places | Whatever you implemented |
| Unit of transfer | A cache line | A page | An object or entry |
| You influence it by | Access pattern and data layout | I/O pattern and hints | Calling it correctly |
When the bet fails
A cache adds latency to a miss, not just speed to a hit — the tag check happens before the miss is even known to be a miss. For workloads with no locality that overhead is real, though small relative to the miss it precedes. The larger cost of a failed bet is capacity: a line that is installed and never reused has evicted a line that might have been.
This is why streaming workloads sometimes benefit from non-temporal stores, which write past the cache rather than allocating a line for data that will never be read again. It is also why "just add more cache" fails for a workload whose problem is that it never revisits anything — the fix for a compulsory miss is not capacity, and Three Kinds of Miss, Three Different Fixes exists to keep those cases apart.
The practical instinct: before optimising for cache, establish that your workload has reuse or adjacency to exploit. If it genuinely does not, the productive move is to change the algorithm so that it does, which is what Cache-Aware Algorithms is about, rather than tuning around a bet that cannot pay.
| Your access pattern has | Can caching help? | The lever that actually applies |
|---|---|---|
| Reuse of the same data | Yes — this is the ideal case | Keep the working set resident (Working Set: Why Performance Falls Off a Cliff) |
| Adjacency but no reuse | Partly — lines still pay for themselves | Streaming is fine; exploit Spatial Locality |
| Reuse, but the set is too large | Not as written | Block the computation so tiles stay resident (Matrix Tiling: Same Arithmetic, Ten Times Faster) |
| Neither reuse nor adjacency | No — more cache changes nothing | Change the algorithm, or accept memory-bound behaviour |
Key points
- A cache holds tagged copies of lines; the authoritative data lives further out, which is why coherence between cores becomes a separate problem.
- Caches are a bet on locality — they convert the regularity real programs happen to have into speed, automatically.
- You cannot place data in a CPU cache from ordinary code; you influence it only through what you access, in what order, and how densely packed it is.
- CPU cache, OS page cache and application cache are three different mechanisms with different managers, units and controls.
- A workload with no reuse and no adjacency cannot be helped by more cache — that is an algorithm problem, not a capacity one.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Core → cache: a load presents an address; the cache extracts an index and a tag from it (Tag, Index and Offset: How an Address Finds Its Line).
- 2Index → set: the index selects a set of candidate slots to search, so only a few tags must be compared rather than all of them.
- 3Tag compare → hit or miss: a matching tag with a valid bit means the copy is present and is returned immediately.
- 4Miss → next level: the request is forwarded outward; meanwhile the core may continue with independent instructions.
- 5Fill → install: the returned line is written into a slot, evicting a victim chosen by an undocumented recency approximation.
- • Believing you can pin a variable in L1 from ordinary source code, and searching for an API that does not exist.
- • Reasoning about the CPU cache using intuitions from an application cache with an explicit
get/putinterface. - • Assuming a bigger cache always helps, when the workload's misses are compulsory rather than capacity-driven.
- • Attributing run-to-run variance to noise when it is residual cache state from whatever ran before (Cache Warmth and the Real Cost of Migration).
Consequences, controls and cost
- • Two runs of identical code differ in speed depending on what else recently ran on that core and left its data behind.
- • Performance changes when unrelated data is added to a structure, because it changed the density of what you actually touch.
- • First-touch costs differ from steady-state costs, so a loop's first iteration is not representative ([[microbenchmarking-pitfalls]]).
- • Cache effects appear as unexplained variance rather than as a clear signal, which is why they are so often misattributed.
- • Make hot data smaller so more of the working set stays resident — this is usually the highest-leverage change available.
- • Access memory in the order it is laid out, so the lines you pay for get used ([[spatial-locality]]).
- • Separate hot fields from cold ones so a fetched line is dense in things you need ([[aos-vs-soa]]).
- • Consider non-temporal stores only for genuine streaming writes, and measure — they are easy to apply wrongly.
- • For workloads with reuse but an oversized set, block the computation rather than tuning anything ([[matrix-tiling]]).
- • Compare cache-reference and cache-miss counters to get a hit rate, rather than reading a raw miss count that scales with work.
- • Run the same function twice back to back and compare — a large first-run penalty indicates cold-cache cost, not code cost.
- • Vary only the size of an unused field in a hot struct; if timing moves, you are observing line density, not logic.
- • Check per-level miss counters to find *where* the misses land, since the fix differs between L1, L2 and last level.
- • Packing data densely for the cache can force awkward types and hurt readability, and sometimes conflicts with alignment requirements ([[alignment]]).
- • Splitting hot and cold fields fragments a natural abstraction and makes the code harder for newcomers to follow.
- • Non-temporal hints improve one pattern and can degrade others badly; they are a measurement-required optimisation, not a default.
Scope
§224 — what these claims are specific to.
- GENERALTagged, hardware-managed, line-granular caching is common to contemporary general-purpose CPUs; hint instructions and their exact semantics are ISA- and microarchitecture-specific
Misconceptions
register in C is a hint about registers that compilers have ignored for decades, and there is no portable construct that pins a line. What you can influence is size, layout and access order — the inputs the cache is deciding on.