Cache Stampede: Everyone Misses at Once
One popular key expires and ten thousand concurrent requests discover the miss simultaneously. Each one dutifully queries the database to repopulate it. The database receives ten thousand copies of the same query, and the cache that was protecting it becomes the mechanism that overloads it.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The mechanism: a miss is not one miss
Cache-aside logic reads "check the cache; on miss, compute and store". Under concurrency that logic contains an unstated assumption — that only one request will find the entry missing. When a popular key expires at 14:03:00, every request arriving in the window between expiry and repopulation finds a miss. If the query takes 400 ms and the key serves 10,000 requests per second, roughly 4,000 requests miss before the first one finishes writing.
Each of those 4,000 issues the same query. The database now has 4,000 concurrent identical requests, which exhausts the connection pool, queues, and pushes query latency up — which extends the repopulation window, which admits more requests into the stampede. The feedback loop is the same shape as Retry Storms: The Load You Generated Yourself: the response to the problem increases the problem.
Three variants are worth recognizing because they have different triggers. Expiry stampede is the classic above. Cold-start stampede happens when a cache node restarts or a deploy invalidates a keyspace — every key misses at once, not just one. Synchronized-expiry stampede is the cruellest: entries written together during a previous incident share a TTL, so they expire together, and the system develops a periodic self-inflicted outage on a TTL-length cycle.
Mitigations, and what each one costs
Single-flight coalescing is the highest-leverage fix and the one to reach for first: on a miss, exactly one caller computes the value while the others wait for that result. It is a lock keyed by the cache key, in-process for a single instance or distributed for a fleet. It converts 4,000 queries into one, and it costs the complexity of a lock with a timeout — because a coalescing lock whose holder dies must not block every subsequent request forever.
Probabilistic early expiry attacks the problem from the other side: instead of all callers treating an entry as valid until the instant it expires, each caller independently and randomly decides to refresh slightly early, with probability rising as expiry approaches. The result is that the entry is usually refreshed by one request before it ever expires, so the miss window never opens. It is elegant and cheap and it does not help at all with cold starts, where there is nothing to expire early.
Serving stale while revalidating asynchronously removes the user-visible impact entirely: the expired value is returned immediately while one background task refreshes it. This is the best user experience of the three and it requires an explicit decision that briefly-stale data is acceptable — which is a correctness question, not a performance one. TTL jitter is the cheapest possible mitigation and should be the default everywhere regardless of what else is chosen: adding a random component to every TTL prevents synchronized expiry from ever developing.
| Mitigation | Mechanism | Handles | What it costs |
|---|---|---|---|
| Single-flight coalescing | One caller computes on miss; others await that result | Expiry and cold start | A lock with a timeout; a dead holder must not block forever |
| Probabilistic early expiry | Each caller may refresh early with rising probability near TTL | Expiry only | Slightly more backend work in steady state; no cold-start help |
| Serve stale, revalidate async | Return the expired value; refresh in the background | Expiry; hides latency entirely | Requires accepting bounded staleness — a correctness decision |
| TTL jitter | Randomize each TTL by ±10–20% | Synchronized expiry | Nearly nothing — this should be the default everywhere |
| Backend concurrency limit | Cap concurrent miss-fills; shed or queue beyond it | All three, as a backstop | Some requests fail fast instead of waiting — deliberate Backpressure |
| Cache warming before traffic | Populate before shifting traffic to a new node | Cold start / deploys | Deploy complexity; needs to know the hot keyset |
| Not a fix | Longer TTLs | Reduces frequency, not severity | A rarer, larger stampede on staler data |
Coalescing, concretely
The code below is the difference between the two behaviors. The first is the textbook cache-aside pattern that appears in most codebases and most tutorials, and it is correct for a single request and catastrophic for ten thousand concurrent ones. The second adds a per-key in-flight map: the first caller starts the work and stores its promise; every subsequent caller finds the promise and awaits the same result.
Two details matter in production and are easy to omit. The in-flight entry must be removed in a finally block, or one failed computation poisons the key permanently. And the wait needs a timeout, so a hung computation degrades to an error for its waiters rather than an unbounded pile-up — which is precisely the queue the coalescing was introduced to prevent.
For a fleet of instances, an in-process map coalesces within each instance, reducing 4,000 queries to one per instance — with fifty instances that is fifty queries instead of 4,000, usually sufficient. A distributed lock reduces it to exactly one across the fleet, at the cost of a network round trip on every miss and a new failure mode when the lock service is unavailable. Start in-process; escalate only if fifty is still too many.
1async function getProduct(id: string) {2 const hit = await cache.get(id)3 if (hit) return hit4 5 // Every concurrent caller that misses arrives here.6 const row = await db.query(EXPENSIVE_PRODUCT_QUERY, [id]) // 400 ms7 await cache.set(id, row, { ttl: 300 })8 return row9}10 11// key expires at 14:03:00, 10k req/s, 400 ms query12// -> ~4,000 identical queries before the first write lands1const inflight = new Map<string, Promise<Product>>()2 3async function getProduct(id: string) {4 const hit = await cache.get(id)5 if (hit) return hit6 7 const existing = inflight.get(id)8 if (existing) return existing // join the in-flight computation9 10 const work = (async () => {11 const row = await db.query(EXPENSIVE_PRODUCT_QUERY, [id])12 await cache.set(id, row, { ttl: jitter(300) }) // jitter: never synchronize13 return row14 })().finally(() => inflight.delete(id)) // never poison the key15 16 inflight.set(id, work)17 return withTimeout(work, 2000) // waiters degrade, they do not pile up18}Both versions produce identical results and identical cache contents. The second issues one database query per instance per expiry instead of thousands, by making concurrent callers share one computation — and the two easily-forgotten details, the finally cleanup and the waiter timeout, are what keep the mitigation from becoming its own failure mode.
Key points
- Cache-aside assumes one caller misses at a time; under concurrency, the whole expiry window misses simultaneously.
- The loop is self-reinforcing: more concurrent misses slow the backend, which widens the window, which admits more misses.
- Three variants — expiry, cold start, synchronized expiry — and no single mitigation covers all of them.
- Single-flight coalescing is the highest-leverage fix; TTL jitter is nearly free and should be the default everywhere.
- Serving stale while revalidating removes user-visible impact and requires an explicit decision that staleness is acceptable.
Cache Hit Rate as a Multiplier
Change an input and watch which number moves — and which one does not.
A 95% hit rate sounds healthy, and it is — until it moves. Dropping from 95% to 90% does not add 5% more database load, it doubles it.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Database → spike: queries for one statement jump from 40/s to 3,900/s within one second, with no deploy and no traffic change.
- 2Cache → correlation: miss rate for a single key prefix spikes in the same second; the aggregate hit rate barely moves.
- 3Application → mechanism: cache-aside code issues the backend query on every miss, so every request in the repopulation window queries independently.
- 4Backend → amplification: the pool saturates, query latency rises from 400 ms to 2 s, extending the window and admitting more requests into the stampede.
- 5Window → root cause: no coalescing on miss, and identical TTLs across the popular keyset, so expiry is both concentrated and synchronized.
- • "Traffic spiked." Request rate was flat; only the miss rate spiked.
- • "The database got slower on its own." The database received a thousand-fold increase in identical queries.
- • "Hit rate looks fine." A three-second stampede does not move a one-minute average hit ratio.
- • "Increase the TTL." Fewer stampedes, each on staler data, and the severity is unchanged.
- • "Add cache memory." Memory addresses eviction, not the expiry-window concurrency that causes this.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Per-statement database query rate at one-second resolution — a stampede is invisible at one-minute granularity.
- • Cache miss rate per key prefix, so a single key's expiry is distinguishable from a general hit-rate decline.
- • Concurrent in-flight backend calls per cache key, which is the number the mitigation is designed to drive to one.
- • Connection pool acquisition wait during the spike, which is usually where the stampede first becomes user-visible ([[connection-pool-saturation]]).
- • Periodicity: if the spike repeats on a fixed interval, it is synchronized expiry and jitter fixes it.
- • Add single-flight coalescing per key, with a `finally` cleanup and a waiter timeout — one computation per instance per expiry.
- • Add jitter to every TTL as a standing default, so synchronized expiry cannot develop after an incident repopulates many keys at once.
- • Serve stale while revalidating asynchronously where the freshness contract permits it, which removes the user-visible latency entirely.
- • Cap concurrent miss-fills against the backend as a backstop, so any future stampede degrades into shed load rather than a collapse.
- • Warm the cache before shifting traffic to restarted or newly deployed nodes, for the cold-start variant that coalescing alone cannot prevent.
- • Reproduce it: force-expire a hot key under load and confirm backend query count per instance is one rather than thousands.
- • Confirm the periodic spike disappeared after adding jitter, observed over several TTL cycles.
- • Check p99 latency for the affected endpoint during a forced expiry, before and after — coalesced waiters still wait, so the improvement should be visible but not total.
- • Verify the waiter timeout works by injecting a hung backend call and confirming waiters fail fast instead of accumulating.
- • Coalescing makes waiters share one computation, so a slow computation makes all of them slow together — the timeout is what bounds that.
- • A distributed lock reduces the fleet to one query and adds a network round trip per miss plus a dependency that can fail.
- • Serving stale trades correctness for latency and needs an owner for the staleness contract ([[cache-invalidation]]).
- • Concurrency limits protect the backend by failing some requests deliberately, which is a reliability trade rather than a free improvement.
- • An alert on per-statement database query rate at one-second resolution, catching spikes a minute-granularity dashboard cannot see.
- • A load test in CI that expires a hot key under concurrency and asserts a bounded number of backend calls.
- • A shared cache helper that has coalescing and jitter built in, so correctness does not depend on each call site remembering.
- • A cold-start drill during deploys, verifying warming works before traffic shifts.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 10,000 req/s, 400 ms query and ~4,000 concurrent misses are a constructed arithmetic example. Real stampede severity depends on key popularity, computation cost and concurrency limits already in place.
- SIMULATEDThe cache simulator attached to this lesson models expiry and coalescing behavior; its output is a model, not a measurement.
- WORKLOAD-SPECIFICWhether coalescing per instance is sufficient depends on fleet size. Fifty instances yields fifty queries; whether that is acceptable depends on backend capacity.