Your Code Is Data Too

The first level of cache is split: a data cache and an instruction cache, each with its own capacity and its own miss behaviour. This split exists because instruction fetch is sequential and read-only while data access is neither, so the two benefit from different design choices. The consequence for a programmer is that code occupies cache in exactly the way data does, competes for capacity in the same way, and suffers the same locality effects.

The instruction cache is typically small — modest by the standards of any data structure. A tight loop that fits inside it can be fetched at full rate indefinitely. A loop whose body has grown past capacity begins evicting its own start before reaching its end, and the front end spends cycles waiting on fetch. Because the working set here is *code size* rather than data size, the usual instincts about locality do not fire.

The unified levels behind it — L2 and beyond — are shared between code and data, which means the two compete. A workload streaming a large array can evict the instruction lines of the very loop doing the streaming, an interaction that is genuinely difficult to reason about from source and shows up only in front-end stall counters.

Inlining removes a real cost: the call sequence, the argument marshalling, the return, and the optimisation barrier a call boundary represents. Having inlined, the compiler can also propagate constants, eliminate dead branches, and vectorize across what used to be a boundary. These are substantial wins and they are why inlining is on by default everywhere.

It also has a real cost, which is code size. Each inlined copy is more bytes of hot instruction stream, and the instruction cache does not grow to accommodate your enthusiasm. In a hot loop calling a small function this is almost always a clear win. In a hot loop calling a large function, or in a call graph where an aggressive inliner has produced many specialised copies, the cost can exceed the benefit and the loop becomes front-end-bound.

The honest summary is that inlining is a size-for-speed trade whose sign depends on the specific code, and compilers make that judgement with heuristics rather than certainty. Which is why forcing it — with attributes, hints or pragmas — is a change that must be *measured* rather than assumed, and one of the few optimisation decisions where the compiler default is usually better than a confident developer.

The signature is distinctive once you know to look for it: poor CPI, low data-cache miss rates, and high front-end stall or instruction-fetch-miss counters. That combination says the core is not waiting on data — it is waiting on the code itself. Because most performance tutorials focus entirely on data, this combination is regularly misread as "unexplained slowness".

Beyond raw code size, three patterns produce it disproportionately. Deeply layered abstractions where every call is a small indirection spread the hot path across many distant lines. Megamorphic call sites, where one virtual call reaches many implementations, defeat both the instruction cache and the branch predictor at once. And templated or generic code instantiated many times generates near-duplicate machine code that multiplies footprint without adding behaviour.

The fixes mirror the data-side ones with the roles swapped: shrink the hot path, move cold code out of line so it stops sharing lines with hot code, reduce needless specialisation, and lay out functions so that code executed together sits together. That last one — profile-guided layout — is one of the few optimisations that is nearly free at the source level, because the compiler and linker do the work once you give them the profile.