Operating a Cache
Hit rate, memory, evictions, hot keys and latency — plus the planning question that decides your real architecture: can the system survive losing the cache?
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
If the cache disappeared right now, would the system stay up?
A cache is added for performance and silently becomes a dependency. The system is then sized for the cached path, and the uncached path — which is the one you get during an incident — has never been tested.
Put a cache in front of the expensive queries. Latency drops, database load drops, everyone is happy. Watch the hit rate occasionally.
The database is now provisioned for the cached load. When the cache is empty, the origin receives traffic it has never handled and falls over (Cache Stampede: Everyone Misses at Once in Backend Engineering).
- The database is now provisioned for the cached load. When the cache is empty, the origin receives traffic it has never handled and falls over (Cache Stampede: Everyone Misses at Once in Backend Engineering).
- A high hit rate hides how expensive the misses are. The average looks excellent while a fraction of users wait for the slow path (Tail Latency: Why p50 Being Fine Does Not Help in Observability).
- Memory fills, eviction begins, and the hit rate degrades gradually — a slow, unalerted decline that ends as a latency incident with no deploy to blame.
- One hot key concentrates traffic on a single node or shard, so the cluster looks unloaded while one member saturates (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability).
- Stale data outlives its usefulness because invalidation was never designed, and users see wrong values with no error anywhere (Cache Invalidation, Stampedes and Hot Keys in Backend Engineering).
- A cache restart or failover empties everything at once, which is the worst possible moment for the origin.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A cache trades memory and staleness for latency and origin load. Every operational signal is about one of those four terms moving.
- Hit rate is a ratio, and ratios hide magnitude. What matters operationally is the miss rate multiplied by origin cost — the load the origin actually receives — which is why a small hit-rate drop on a large volume is a big event.
- Memory pressure produces eviction, eviction produces misses, and misses produce origin load. The chain is why memory headroom is a reliability signal and not a cost signal.
- Hot keys break the assumption that load distributes: a single key cannot be sharded, so its traffic lands on one node no matter how many you add (Consistent Hashing in Architecture).
- Simultaneous misses on the same key stampede the origin. Single-flight, request coalescing, or serving stale while refreshing are the standard defences, and they are application behaviour rather than cache configuration.
- Cache loss is the operational scenario that matters most, because it is the one that converts a performance component into an availability event.
Five signals, and which one leads
Hit rate is the metric everyone watches and the last one to move. Memory and evictions move first, which makes them the alertable pair.
- Alert on memory headroom and eviction rate; watch hit rate as context.
- Track origin load alongside cache hit rate on the same dashboard — that pairing is what makes a hit rate change legible.
- A hit rate that is suspiciously stable often means a metric that stopped updating.
| Signal | What it tells you | Why it matters operationally |
|---|---|---|
| Hit rate | Fraction of reads served from cache | A ratio — must be read together with volume, because origin load is misses times cost |
| Memory used vs limit | Whether your TTL policy is still in effect | Leading signal: pressure precedes eviction precedes misses |
| Eviction rate | The cache discarding entries you wanted kept | Non-zero means the cache is deciding your policy for you |
| Hit and miss latency, separately | What each path costs | An averaged latency graph hides a slow miss path entirely (Tail Latency: Why p50 Being Fine Does Not Help in Observability) |
| Key / shard distribution | Whether load is spread | One hot key saturates one node while the cluster looks idle (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability) |
The question that decides the architecture
Cold cache is not a hypothetical. It happens on failover, restart, deploy of a new key format, region change, or an emergency flush. What matters is which of these answers is honestly yours, because each implies different capacity and different code.
The uncomfortable case is the third: it is extremely common, usually unintentional, and almost never written down anywhere.
The cache is empty right now and full production traffic is arriving. What happens?
when Origin capacity covers the full uncached load. The cache is a genuine optimisation.
cost Paying for origin capacity that is idle while the cache is healthy — the price of the cache being optional (Headroom).
when Concurrency limits, load shedding and stale-while-revalidate keep the origin alive at reduced quality.
cost Application complexity: single-flight, shedding policy, and a defined degraded experience someone has agreed to (Load Shedding).
when This is the default when nobody asked the question, and it is where most systems actually are.
cost The cache is now a tier-one dependency with no acknowledged availability requirement, no runbook and usually no redundancy. Recognising this is the point of the lesson.
when Traffic is held or shifted until the cache is populated, then released.
cost Longer recovery, extra orchestration, and a warming path that must itself be maintained and rehearsed.
Cache incidents and their responses
Note how many of these are caused by a well-intentioned operator action. The cache is the component where the instinctive fix is most likely to be the incident.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Cache node restarts or fails over | Origin load spikes; latency and errors climb together | Cold cache with an origin sized for the cached path | Concurrency limit on origin calls; warm before full traffic; size for the uncached case |
| Popular key expires | A burst of identical origin requests | Stampede — many concurrent misses on one key | Single-flight or request coalescing; serve stale while one request refreshes (Cache Stampede: Everyone Misses at Once in Backend Engineering) |
| Bulk write populated many keys at once | Periodic synchronised load spikes | Identical TTLs expiring together | Jitter expiry times per key |
| One item becomes viral | One cache node saturated; cluster average low | A hot key cannot be sharded | Replicate that key across nodes or add a short-lived local cache in front (Hot Keys: When Aggregate Metrics Hide a Saturated Node in Observability) |
| Operator flushes to clear stale data | Immediate origin overload | Global flush during peak | Invalidate the affected key space only; treat a global flush as an incident-level action |
| Origin returns an error and it is cached | Errors served long after the origin recovered | Negative caching without a short, separate TTL | Cache failures briefly and deliberately, or not at all |
| Cache unreachable from the application | Application returns 500s although the origin is healthy | Client treats cache failure as an error rather than a miss | Short timeout on cache calls; on failure, fall through to the origin |
How to do it properly
Most important first.
- Answer the cold-cache question explicitly for each cache: can the origin serve the full uncached load, at reduced quality, or not at all? Write the answer down; it is an architectural fact, not a hope.
- Watch five signals: hit rate, memory used against limit, eviction rate, latency of both hit and miss paths, and key or shard distribution.
- Alert on eviction rate and memory headroom, because those lead the hit-rate decline that leads the incident.
- Defend the origin: single-flight on misses, jittered expiry so keys do not expire together, and a concurrency limit on origin calls so a stampede degrades rather than collapses.
- Prefer serving stale data while refreshing in the background where the data tolerates it — this converts an availability problem into a freshness problem.
- Design invalidation with the write path, not afterwards; a cache whose invalidation is a scheduled sweep is a cache that serves wrong data by design.
- Keep the cache optional in code: a failure to reach it must be a miss, never an error (Reliability Patterns in Architecture).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Client-side degradation contains it — cache errors treated as misses, origin concurrency limits, stale-while-revalidate. Without those, a cache incident is an origin incident within seconds.
What can go wrong
- Cache failure surfaces as application errors because the client treats an unreachable cache as an exception rather than a miss.
- A mass expiry — keys written together with the same TTL — empties a whole class of entries simultaneously.
- A flush issued to fix stale data takes the origin down.
- A hot key saturates one node while the cluster reports low average utilisation.
- Caching an error response or an empty result, so a transient failure is served for the TTL duration.
- "Hit rate is 95%, the cache is healthy." The 5% may be the expensive path, and a fall to 90% doubles origin load (A 95% Hit Rate Tells You Almost Nothing in Observability).
- "The cache is just an optimisation." It is an optimisation until the origin is sized around it. After that it is a dependency with an availability requirement.
- "Flush the cache" as a routine fix. On a large key space during peak, a flush is a self-inflicted outage.
- "More cache nodes fix the hot key." A single key lives on one node; adding nodes does not split it. Replication of that key, or a local cache in front, is the actual answer.
- "Caching makes the system faster." It makes the hit path faster and the miss path slightly slower, and it makes the system's worst case worse (When Not to Cache in Backend Engineering).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A load test or a real event in which the cache was cold and the system's behaviour was observed, not predicted.
- Dashboards showing hit rate, memory headroom, eviction rate, and hit versus miss latency separately.
- Alerts on eviction rate and memory that have a documented response.
- Key distribution visibility good enough to name the top keys during an incident.
- Removing a cache is easy in code and dangerous in production: the origin must be able to carry the load, which is the same question as surviving cache loss.
- A flush is not reversible. The correct emergency lever is usually targeted invalidation of the affected key space, not a global flush.
- If a cache change goes wrong, the safest step is often to serve stale rather than to empty — stale data is a smaller failure than an unavailable origin.
- Automate: memory and eviction alerting, hot key detection, expiry jitter, single-flight on misses, and warming after a planned restart.
- Keep human: flushing production caches, changing TTL policy for a high-volume key space, and deciding whether stale data is acceptable for a given field.
- Longer TTLs raise hit rate and increase staleness; shorter TTLs keep data fresh and push load to the origin. There is no setting that is correct for all keys, which is why TTL belongs per key class.
- A local in-process cache is fastest and multiplies staleness by instance count; a shared cache is consistent and adds a network hop and a shared failure domain (Local vs Distributed Cache in Backend Engineering).
- Sizing the origin to survive a cold cache costs capacity that is idle whenever the cache is healthy — which is almost always (Headroom).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICEviction policy, memory accounting and clustering differ by product. A Redis-style store gives you configurable eviction policies and per-key visibility; a CDN or HTTP cache gives you headers and purge APIs and very little introspection. What you can observe and what you can invalidate differ far more than the caching concept does.
- GENERALThe five signals and the cold-cache question apply to any cache layer: in-process, shared, database buffer pool or CDN edge. The remedies differ; the questions do not.
- SCALE-SPECIFICBelow the volume where the origin cannot serve everything, a cache is genuinely optional and its loss is a latency event. Above it, the cache is part of the availability story and needs the same rigour as the database.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — where a cache belongs in a request path, and how caching at the edge, in the application and in the database interact.