The question this answers
Why does one small event produce a load spike ten thousand times its size, and how do I stop the crowd forming?
10,000 in-flight requests waiting on a cached homepage payload whose TTL expires at exactly 12:00:00, plus 4,000 clients that reconnect simultaneously when a websocket gateway restarts.
The cache entry everyone is reading, the origin that regenerates it, and the connection pool between them. The herd itself is emergent: nobody coordinates it, and the coordination is the problem — everyone arrives at once because everyone was released at once.
Concurrent demand on the shared resource stays bounded regardless of how many tasks are waiting, and the work behind a single event is performed once rather than once per waiter.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Ten thousand waiters, one release, one millisecond
A herd forms whenever a large number of tasks are blocked on the same condition and that condition becomes true for all of them at once. The classic sources are all mundane: a cache key expiring, a lock being released with many waiters, a database coming back after a failover, a gateway restarting so every client reconnects, and a cron schedule that a thousand instances all interpret as exactly :00:00.
What makes it dangerous is that the arrival pattern, not the total work, is the problem. Ten thousand cache regenerations spread over sixty seconds is nothing; ten thousand in one millisecond exhausts the pool, saturates the origin, and pushes every request past its timeout — after which the clients retry and it happens again, larger. The system has a stable state and a collapsed state, and the herd is the kick that moves it from one to the other.
The specifically cache-shaped version of this is a stampede, and the performance domain covers its diagnosis in cache-stampede. This lesson is about the general shape and the coordination primitives that address it.
The schedule, and why the retry makes it worse
The step that turns an overload into an outage is the retry. Every one of the ten thousand requests has a client with a retry policy, and those policies were written independently by people who never imagined ten thousand of them firing together. Without jitter on the retry, the second wave is not merely as large as the first — it is *more* synchronised, because the timeouts all fired at the same moment.
That is the general principle behind every mitigation here: desynchronise, or deduplicate. Jitter desynchronises by spreading arrivals in time. Single-flight deduplicates by making the work happen once. Batching does both. A concurrency limit does neither but caps the damage. They compose, and in a serious system you use several.
| # | Cache | 10,000 requests | Origin service | Clients (retry policies) | State |
|---|---|---|---|---|---|
| 1 | key homepage:v3 expires at 12:00:00.000 | · | · | · | entry=absent originConc=0 served=all |
| 2 | · | all 10,000 in-flight requests miss within ~2 ms | · | · | entry=absent originConc=10000 served=none |
| 3 | · | · | accepts what the pool allows; 9,980 queue for a connection | · | entry=absent originConc=20 queued=9980 ✕ One expiry produced 10,000 identical regenerations of the same value. 9,999 of them are pure waste even if they all succeed. |
| 4 | · | · | p99 crosses the 2 s request timeout; queued requests start failing | · | entry=absent originConc=20 queued=9980 failing=yes |
| 5 | · | · | · | retry policies fire — most with a fixed backoff, no jitter | entry=absent originConc=20 queued=19000 ✕ The retry wave is larger AND more synchronised than the original, because every timeout expired at the same instant. The system cannot drain and will not recover on its own. |
| 6 | a regeneration finally succeeds and fills the key | · | · | · | entry=present originConc=20 queued=12000 |
Four mitigations, and what each one costs
These are not alternatives to choose between — they address different parts of the problem and compose well. Jitter prevents the herd from forming; single-flight collapses it after it forms; batching amortises it; a concurrency limit contains whatever gets through. A cache in front of a slow origin usually wants all four.
The one to be careful with is the concurrency limit on its own. It caps offered load, which prevents the origin from falling over, but the excess has to go somewhere: it waits, or it is rejected. Waiting without a bound is how a queue becomes the new failure (Bounded vs Unbounded Queues); rejecting is honest and requires the caller to handle it. Decide which, explicitly.
| Mitigation | Mechanism | Reduces | Cost / caveat |
|---|---|---|---|
| Jitter on TTLs | Randomise expiry: ttl * (0.8 + random() * 0.4) | Arrival synchronisation | Entries live slightly longer or shorter than configured; not applicable when the invalidation is event-driven |
| Jitter on retries | Randomised (ideally full-jitter) exponential backoff | The second and every later wave | Longer worst-case recovery for an individual client; must be in every client, and usually is not |
| Single-flight | One in-flight fetch per key; everyone else subscribes | Duplicate work, from N to 1 | A shared failure: everyone gets the same error. Needs careful cleanup (Single-Flight Coalescing) |
| Early / probabilistic refresh | Refresh before expiry, one lucky request at a time | Misses entirely — the entry never goes cold | Extra background load, and a stale window while refreshing |
| Batching | Collect requests for a few ms and issue one combined call | Request count, from N to N/batch | Adds the batch window to every request's latency; needs a batch-capable downstream |
| Concurrency limit / semaphore | At most K concurrent calls to the resource | Peak offered load | The excess waits or is rejected — pick one; an unbounded wait queue is the next incident |
| Serve stale while revalidating | Return the expired value, refresh in the background | The user-visible impact of a miss | Staleness becomes a product decision, not an implementation detail |
| Desynchronise schedules | Per-instance offset instead of :00:00 | Cron and heartbeat herds | Job start times become non-deterministic, which complicates reasoning about ordering |
Key points
- A herd is an arrival-pattern problem, not a total-work problem: the same work spread over a minute is invisible and in one millisecond is an outage.
- The triggers are mundane — a TTL expiring, a lock released, a gateway restarting, a cron at
:00:00, a failover completing. - Retries without jitter make the second wave larger and *more* synchronised than the first, which is why the system does not recover on its own.
- Two families of fix: desynchronise (jitter, staggered schedules, early refresh) or deduplicate (single-flight, batching).
- A concurrency limit contains the damage but does not remove the demand — decide explicitly whether the excess waits or is rejected.
- Jitter belongs on the retry path as much as on the TTL; most retry policies in most clients have none.
- These mitigations compose, and a cache in front of a slow origin usually wants several of them.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Many tasks block on the same condition — a key's presence, a lock, a connection, a clock.
- • The condition becomes true for all of them simultaneously, and every waiter is made runnable in the same instant.
- • Each one independently performs the same next action against the same resource, so offered concurrency jumps from near zero to N.
- • The resource's queue forms; service time degrades; requests approach and then exceed their timeouts.
- • Timeouts fire — all at roughly the same moment, because they all started at the same moment — and clients retry.
- • The retry wave arrives even more tightly synchronised than the original, and the resource has not drained, so the second collapse is faster than the first.
- • Recovery requires either external intervention, load shedding, or one request finally succeeding and refilling whatever everyone was waiting for.
- • 12:00:00.000 the key expires; by 12:00:00.002 all 10,000 in-flight requests have missed; the pool admits 20 and 9,980 queue for a connection.
- • At 12:00:02 the request timeout fires for the queued requests — all of them within a few milliseconds of each other, because they all started together — and every client retries.
- • The retry wave arrives at 12:00:02.5, 19,000 strong and more tightly clustered than the first; the origin has not drained a single request.
- • With single-flight: the first miss starts one origin fetch and 9,999 later misses attach to the same in-flight handle; the origin sees exactly one request; the result is broadcast and the cache is filled.
- • With jittered TTLs: expiries are spread over 60 seconds, so the origin sees roughly 170 regenerations per second, which is ordinary traffic.
- • Lock release with 500 waiters: all 500 become runnable, all 500 contend for the same lock, 499 immediately block again — a wake storm that costs 500 context switches to accomplish one acquisition (What Contention Actually Costs).
- • Gateway restart: 4,000 websocket clients reconnect within a second with no backoff, and the authentication service — which was never the problem — is the component that fails.
- • Guaranteed: jitter spreads arrivals in time. It does not reduce total work by a single unit.
- • Guaranteed: single-flight reduces concurrent duplicate work for one key to one, for as long as the in-flight handle exists.
- • Guaranteed: a concurrency limit caps offered load at K, whatever the demand.
- • NOT guaranteed: that a concurrency limit protects you. The excess still exists — as a queue that can grow without bound, or as rejections.
- • NOT guaranteed: that jitter helps an event-driven invalidation. If the trigger is a publish rather than a clock, there is no TTL to randomise.
- • NOT guaranteed: that single-flight helps across processes. It is per-process unless the coordination is shared, and twenty instances means twenty flights (A Mutex on Server A Does Nothing About Server B).
- • NOT guaranteed: recovery once retries are compounding. The system may need shedding to return to a stable state.
- • The awaited resource is contended by every waiter simultaneously — a pool, an origin, a lock, an auth service.
- • Wake-up itself is contention: N runnable tasks contend for cores and for the cache lines they are all about to touch.
- • On a lock, the herd produces N−1 wasted wakeups per release, each one a context switch that accomplishes nothing (Lock Convoys).
- • The queue in front of the resource is the second contention point, and an unbounded one converts an overload into a memory problem.
- • Retries add contention precisely when the resource has the least capacity to absorb it.
- • Cache stampede: N identical regenerations of one value, of which N−1 are pure waste.
- • Pool exhaustion, with the blast radius covering every endpoint sharing the pool (Parallelism Moves the Load Downstream).
- • Retry storm: successive waves larger and more synchronised than the first, preventing recovery (Retry Storms: The Load You Generated Yourself).
- • Wake storm on a lock or condition variable: N wakeups, one winner, N−1 immediate re-blocks.
- • Reconnect storm after a restart, where the authentication or session service fails rather than the gateway that restarted.
- • Cron convergence: a thousand instances all scheduled at
:00:00producing a load spike every hour, on the hour, forever. - • Self-inflicted synchronisation: a fix that resets all TTLs at once — such as a cache warm-up on deploy — recreating the herd on a new schedule.
- • Nothing about a herd helps, but the *waiting* usually does: many tasks blocked on one condition is a normal, efficient design.
- • A deliberate herd is occasionally correct — releasing all participants at a barrier, or starting a load test with a synchronised start latch.
- • The mitigations help most where the awaited work is expensive and identical across waiters, which is the cache case exactly.
- • Jitter is nearly free and should be the default on every TTL and every retry policy, whether or not you currently have a herd.
- • Any time the waiter count is large and the awaited resource has a hard capacity ceiling.
- • Any time retries are synchronised, which is the default in most clients unless someone explicitly added jitter.
- • Any scheduled work expressed as an exact clock time across many instances.
- • Any invalidation that clears many keys at once — a deploy-time cache flush is a herd generator with excellent intentions.
- • Any system with an unbounded wait queue, where the herd becomes memory exhaustion rather than rejection.
- • Concurrent requests to the resource at the moment of the event — a spike from ~1 to N is the whole diagnosis.
- • Cache miss rate as a time series with sub-second resolution; a stampede is a single-bin spike that a 60-second average erases completely.
- • Ratio of origin requests to cache misses. Single-flight should drive it far below 1; if it is 1, the coalescing is not working.
- • Retry counts as a proportion of total requests, and the inter-arrival distribution of retries — clustering is the signal that jitter is missing.
- • Pool acquisition wait time at p99 during the event (Connection Pool Saturation: Waiting in Front of an Idle Database).
- • For scheduled work: a histogram of job start times. Every instance at
:00:00.000is visible instantly and fixed with a per-instance offset.
- • Jitter makes behaviour non-deterministic, which is exactly the point and does complicate reproducing a bug.
- • Single-flight adds an in-flight map with a lifecycle, a dedup key design and a failure policy (Single-Flight Coalescing).
- • Batching adds a window that must be tuned and that appears in every request's latency budget.
- • Concurrency limits add a bound to justify and a rejection path the caller must handle.
- • Several mitigations composed means several interacting parameters, and their interaction is the thing nobody documents.
- • Never let the entry go cold: refresh in the background before expiry, so there is no miss to stampede on.
- • Serve stale while revalidating, which removes the user-visible impact and turns staleness into an explicit product decision.
- • Make the work cheap enough that N copies do not matter — sometimes the real fix is a faster origin, not less concurrency.
- • Push the value instead of pulling it: publish updates into the cache so it is never invalidated by a clock at all.
- • A shared coordination point (a distributed lock or a leader) when single-flight must span processes — with the warning in A Mutex on Server A Does Nothing About Server B very much attached.
- • Load shedding at the edge, which is the honest answer when demand genuinely exceeds capacity and no amount of spreading changes that.
Thundering herd: 10,000 waiters
no mitigation 10K requests in one 100 ms bucket vs a capacity of 90/bucket → 9,640 shed this setting 10K requests spread over one instant → peak 10K/bucket, 9,640 shed jitter sleep(base + random() * window) — decorrelates wakeups; costs a little latency batching one call serves N waiters — cuts the request count, not the wakeup count both are cheaper than the capacity you would otherwise have to buy for one instant per hour
Single-flight: N callers, one call
without 64 callers → 64 downstream calls latency 192 ms (queued behind each other) with 64 callers → 1 downstream call latency 60 ms (everyone waits for the leader) failure one attempt, 64 disappointed callers — the blast radius of a single bad call is now N retry the followers cannot retry independently; they only ever saw the leader's outcome
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
Eight threads, one lock
What people believe, and what is true
It is only 10,000 requests — we handle more than that per second.
You handle them spread across a second. Ten thousand in one millisecond is an instantaneous concurrency of 10,000, and shared resources are sized for concurrency.
We added retries, so transient overload is handled.
Retries without jitter are the amplifier. Every timeout fired at the same instant, so the retry wave is more synchronised than the original and the system cannot drain.
A concurrency limit fixes it.
It caps what reaches the resource. The demand still exists and now sits in a queue — which, if unbounded, is the next incident with a different name.
Go deeper
Overview
One event releases a crowd, and the crowd all does the same thing to the same resource at the same time.
Practical
Jitter every TTL and every retry, single-flight expensive shared work, cap concurrency to the resource, and never schedule anything at exactly :00:00 across many instances.
Advanced
The retry loop is what converts an overload into a metastable failure: the second wave is larger and better synchronised than the first, so the system has a stable state and a collapsed state and no path between them without shedding.
Internals
At the primitive level this is notify_all versus notify_one, and a futex wake count. Waking N threads to have one acquire a lock costs N context switches and N cache-line transfers to accomplish a single acquisition — the same shape, four orders of magnitude smaller.