C++ Memory Performance: Allocation, Copies and Locality

RAII gives C++ something managed runtimes do not have: destruction happens at a known point, so there is no collector, no pause and no heap headroom to size. That is a genuine advantage for tail latency, and it is why the failure modes here look nothing like Garbage Collection: Pause, Throughput, Footprint — Pick Two.

What replaces it is allocator behaviour. new and delete are not free: a general-purpose allocator maintains size classes and free lists, may take a lock under contention, and occasionally asks the kernel for more memory. Under multi-threaded load, allocator contention becomes a real and easily-missed bottleneck — a profile showing significant self time inside malloc is usually a signal to allocate less rather than to change allocators, though switching to a thread-caching allocator is often a cheap first win.

Fragmentation is the other half, and it is why "resident memory grows and never returns" happens without any leak. Freed blocks that are too small or badly placed to satisfy later requests leave the heap holding memory it cannot reuse. The process is not leaking — every allocation has a matching free — and RSS still climbs. Distinguishing this from a true leak requires allocator statistics, not just the process memory graph (Memory Leaks: Growth That Does Not Come Back, Leak or Unbounded Cache? The Question That Picks the Fix).

C++ will copy for you silently, and each copy of a container is an allocation plus a memcpy. Passing a std::vector by value to a function that only reads it, returning by value in a way that defeats elision, storing into a container without reserve, or capturing by value in a lambda — each is a line that looks free and is not.

Move semantics exist to make transfer of ownership cheap, but only when the code lets them apply. std::move on a const reference silently does nothing. A copy into a container that is about to grow will still reallocate and copy everything. And a function taking const std::string& called with a string literal constructs a temporary anyway. None of these are exotic bugs; they are ordinary lines that a profile flags and a reading of the source does not.

The reason this matters for a *performance* domain rather than a C++ style guide: allocation and copy cost is the C++ equivalent of allocation pressure in a managed runtime. It is diagnosed the same way — profile, find the hot allocation site, remove the allocation rather than optimizing around it — even though the mechanism underneath is completely different.

Modern processors are far faster than main memory, so the practical cost of an operation depends heavily on whether its data is already in cache. A loop over a contiguous array and a loop over a linked list with the same number of elements and the same asymptotic complexity can differ by a large factor, entirely because one access pattern is prefetchable and the other is a chain of dependent loads.

This is the concrete version of the caveat attached to Algorithmic Cost in a Request Handler: big-O counts operations, and operations are not the unit of cost. For small n, an O(n) scan over a contiguous vector routinely beats an O(log n) lookup in a pointer-chasing tree, because the scan touches a handful of cache lines and the tree touches a handful of cache *misses*. Where the crossover sits depends on the hardware, the element size and the access pattern, which is why this is a measurement rather than a rule.

The design consequences are structural: prefer contiguous containers by default, consider struct-of-arrays when a hot loop touches one field of many, and treat working-set size as a first-class quantity. Verify with hardware counters (perf stat cache-miss rates) rather than reasoning, because intuition about locality is unreliable and the counters are cheap to read.