Request Coalescing
When N concurrent requests need the same expensive result, do the work once and share it — the in-process answer to a stampede.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
A thousand requests all miss the cache for the same key at the same moment. Do I really run that query a thousand times?
The homepage shows a computed feed that takes seconds to build. It is cached, but when the cache entry expires every visitor arrives at once.
Cache-aside: on a miss, compute the value and store it. Each request checks the cache, and the ones that miss do the work.
The cache entry expires and, in the window before the first computation finishes, every concurrent request also misses. A thousand identical expensive computations start (Cache Stampede).
- The cache entry expires and, in the window before the first computation finishes, every concurrent request also misses. A thousand identical expensive computations start (Cache Stampede).
- They all hit the database simultaneously, so the query that is fine once per minute is now a thousand concurrent copies, and the pool is gone (Connection Pool Exhaustion).
- The database slows under that load, so each computation takes longer, so the window widens and more requests pile in. The stampede is self-reinforcing (Thundering Herd).
- When they finally finish, all thousand write the same value to the cache — wasted work, and a write burst on the cache.
- A popular key makes this worse precisely because it is popular: the more traffic a key has, the more requests arrive inside the recompute window (Hot Keys: When Aggregate Metrics Hide a Saturated Node).
What is actually happening
- Coalescing — also called single-flight or request deduplication — keeps an in-process map from key to the in-flight promise or future computing that key. The first caller starts the work; subsequent callers find the entry and await the same result (Single-Flight Coalescing).
- The map entry must be inserted before the work starts and, critically, in the same synchronous step as the lookup. If there is an
awaitbetween checking the map and inserting into it, two callers can both check and both start (Initialization Races). - It is per process. Ten instances coalescing independently produce ten computations rather than a thousand — a hundredfold improvement that is still not one (A Mutex on Server A Does Nothing About Server B).
- For one computation across all instances you need a shared lock or a shared claim, which reintroduces a network round trip and a lock-expiry problem. Whether that is worth it depends on how expensive the work is.
- Coalescing is distinct from caching. The cache stops repeated work over *time*; coalescing stops repeated work at the *same moment*. They solve adjacent problems and are usually deployed together (Cache-Aside).
- The alternatives to coalescing address the same stampede differently: probabilistic early recomputation refreshes before expiry so the herd never forms, and serving stale while revalidating in the background means a miss never blocks (TTL and Expiry).
One flight, many passengers
get and set no other JavaScript runs. In Java the equivalent is ConcurrentHashMap.computeIfAbsent, in Go the singleflight package or a mutex-guarded map, in Python with threads a dict guarded by a lock. The comment about the suspension point applies to any async runtime, including Python asyncio.The mechanism is a map from key to in-flight work. The first caller for a key finds nothing, creates the promise, and stores it. Every caller arriving while it is unresolved finds the promise and awaits it. When it settles, everyone resumes with the same value and the entry is removed.
The subtle requirement is that the lookup and the insertion happen with nothing in between. On a single-threaded event loop that means no await between them — which sounds trivial and is violated the moment someone adds an asynchronous cache check inside the coalescing function. On a threaded runtime it means an actual atomic map operation.
The second requirement is cleanup in finally. If a rejected promise is left in the map, every later caller for that key awaits a promise that has already failed, and the key is dead until the process restarts — a bug that survives deploys only by accident.
1const inFlight = new Map<string, Promise<Feed>>()2 3export function getFeed(key: string): Promise<Feed> {4 // Check and insert with NO await between them. On one loop thread5 // nothing can interleave here, so two callers cannot both be leaders.6 const existing = inFlight.get(key)7 if (existing) {8 metrics.increment('coalesce.joined')9 return existing10 }11 12 const flight = computeFeed(key)13 .finally(() => inFlight.delete(key)) // removed on success AND failure14 15 inFlight.set(key, flight)16 metrics.increment('coalesce.started')17 return flight18}19 20// WRONG, and it looks almost identical:21//22// const cached = await cache.get(key) // <- suspension point23// if (cached) return cached24// const existing = inFlight.get(key) // two callers reach here25// ...26//27// The await hands control back to the loop, a second request runs the28// same lines, and both become leaders. Do the cache read before the29// coalescing function, or inside the flight.The commented-out version is the one that gets written in review-passing code. The map is correct; the suspension point above it is what breaks it.
Coalescing, caching and locking are three different things
These get conflated because they all reduce repeated work, and they behave very differently under a stampede. A cache prevents work that has already been done from being done again. A lock prevents two workers proceeding at once — but the second worker, once it acquires the lock, still does the work unless it re-checks. Coalescing means the second caller never does the work at all; it receives the first one's result.
That distinction is what makes coalescing the right tool for a cache miss on a hot key. A lock around the recompute serialises a thousand requests into a thousand sequential computations unless each re-checks the cache after acquiring — which is the double-checked pattern, and is easy to get subtly wrong (Double-Checked Locking: The Canonical Cautionary Tale).
In practice the layered answer is: refresh ahead of expiry so misses are rare, coalesce so a miss costs one computation per instance, and serve stale while revalidating so no caller waits at all.
| Mechanism | What the second caller does | Scope | Best against |
|---|---|---|---|
| Cache | Reads the stored value | Shared, if the cache is | Repeated work over time |
| Coalescing / single-flight | Awaits the first caller's result | One process | Concurrent identical misses |
| Lock + re-check cache | Waits, then finds the value cached | Process or distributed | Stampedes when a shared cache exists |
| Lock without re-check | Waits, then does the work anyway | Process or distributed | Nothing — this is the common mistake |
| Early / probabilistic refresh | Never misses; refresh happened ahead of expiry | Per instance | Preventing the herd forming at all (TTL and Expiry) |
| Stale-while-revalidate | Gets the stale value immediately | Shared | Latency during recompute |
| Distributed lock | Waits or serves stale | All instances | Very expensive computations (A Mutex on Server A Does Nothing About Server B) |
What coalescing does not solve
It is worth being clear about the boundary. Coalescing collapses concurrent identical reads within one process. It does nothing about the same key being recomputed on ten instances, nothing about the next expiry, and nothing at all about writes.
The instance multiplication is the one people are surprised by. Coalescing a thousand concurrent requests down to one per process still means ten computations across ten instances, and if that computation is heavy enough to hurt the database, ten is still too many. The fix is either a distributed claim — with the lock-expiry problem that brings — or staggering expiry so the instances do not all miss at the same instant.
Writes are the other boundary, and the rule is simple: two callers asking to *do* something are two intents, not one question. Deduplicating them is the job of idempotency keys, which are explicit about identity precisely because the server cannot infer it (Idempotency Keys).
How expensive is one computation, and how many instances miss simultaneously?
when The computation is a second or two and the instance count is modest.
cost Origin load proportional to instance count on every expiry.
when Many instances cache the same keys with the same TTL.
cost Slightly different data across instances during the spread window (TTL and Expiry).
when A predictable hot key that must never be cold.
cost Work done for values that may not be requested; a refresh path to maintain.
when The computation is expensive enough that N is genuinely harmful.
cost A shared store on the read path; a lock TTL that must exceed the work, or two leaders appear.
when The value is expensive and its inputs change on a known schedule.
cost A job, a schedule, and staleness bounded by the interval (Scheduled Jobs).
How to build it
Most important first.
- Key the in-flight map by exactly what determines the result. Too coarse and callers receive someone else's data; too fine and nothing coalesces (Where the Check Belongs matters here — a per-user result must be keyed per user).
- Insert into the map synchronously, before any
await. The check and the insert must not be separated by a suspension point. - Always remove the entry in a
finally, on success and on failure alike, or one error poisons the key permanently. - Decide whether followers share a failure. Sharing is usually right — otherwise a failed computation is retried by a thousand callers at once — and it means one transient error is returned to everyone.
- Give the shared computation a timeout, so followers are not blocked indefinitely by one stuck leader (Timeouts).
- Combine with a stale-while-revalidate cache so that a miss is rare in the first place; coalescing then handles the genuinely cold case (Cache Stampede).
- For cross-instance coalescing, use a short-lived lock with a TTL comfortably above the work duration, and decide explicitly what non-holders do: wait briefly, serve stale, or fail fast.
What can go wrong
- A gap between the map check and the map insert — an
awaiton the cache read, for instance — which allows two leaders and defeats the mechanism entirely. - The entry not removed on the error path, so every subsequent caller awaits a promise that has already rejected.
- Keying too coarsely, so a per-user computation is shared across users — a correctness and authorization bug, not a performance one.
- Unbounded growth of the in-flight map when keys are user-controlled, which is a memory leak with an attacker-controlled rate (Resource Limits).
- Followers waiting with no timeout, so one stuck leader stalls every caller for that key.
- A distributed lock whose TTL is shorter than the computation, producing two leaders and no error.
- Coalescing writes rather than reads, which is almost always wrong: two callers asking to change something are not asking the same question.
- Two callers checking the in-flight map before either inserts — the race the mechanism must avoid in its own implementation (Initialization Races).
- A leader completing and removing its entry just as a new caller checks, causing a second computation. Harmless, and the reason the coalesce ratio is never perfect.
- A distributed lock expiring mid-computation, producing two leaders across instances.
- Cache invalidation racing a coalesced computation, so the shared result is written after an invalidation and is already stale (Cache Invalidation).
- Followers attached to a promise that rejects, all resuming simultaneously and all retrying at once unless the retry is bounded (Thundering Herd).
- The key must include every input that affects authorization. Coalescing a per-tenant query on a key that omits the tenant returns one tenant's data to another (Multi-Tenancy).
- Followers must still pass their own authorization checks. Sharing the computation must not mean sharing the permission decision (Object-Level Authorization).
- An in-flight map keyed by user input is an unbounded allocation channel; cap its size and reject beyond it.
- Timing differences between a coalesced follower and a leader can leak whether a key is currently being computed. Rarely significant, and worth knowing where key existence is itself sensitive.
- "The cache handles this." A cache prevents repeated work across time. During the miss window there is nothing cached, and that window is exactly when the stampede happens (Cache Stampede).
- "Coalescing gives one computation." One per process. With ten instances, ten (A Mutex on Server A Does Nothing About Server B).
- "It is just a lock." A lock makes followers wait and then do the work themselves. Coalescing makes them wait and receive the leader's result — no repeated work at all.
- "Coalesce everything." Only idempotent reads. Two callers asking to create an order are not one request (Idempotency in Backends).
- "A promise map is trivially safe." It is safe only if the check and the insert are not separated by a suspension point, which is easy to violate accidentally (Initialization Races).
Operating it
- Coalesce ratio: followers joined divided by computations started. A ratio near one means coalescing is not helping — either keys are too granular or arrivals are spread out.
- In-flight map size as a gauge, with an alert on growth. It should return to zero between bursts.
- Leader computation duration against follower wait time. A large gap means followers are waiting on something other than the computation.
- Origin load per key — the metric that shows the stampede is gone is the count of database queries for that key during an expiry, which should be one per instance (A 95% Hit Rate Tells You Almost Nothing).
- Coalescing gets more valuable as traffic rises, because more requests arrive inside each recompute window. It is one of the few mechanisms whose benefit grows with load.
- Per-instance coalescing divides origin load by the number of concurrent requests per instance; adding instances reduces the benefit, since each has its own map.
- At high instance counts, cross-instance coordination starts to matter, and the choice becomes a shared lock, staggered expiry, or accepting N computations for N instances.
- For the very hottest keys, coalescing alone is insufficient and the answer is to stop expiring them synchronously at all — refresh ahead of expiry so no request ever finds a cold key (TTL and Expiry).
- Coalescing couples callers: they share a result, a latency and a failure. One slow computation is now slow for everyone waiting on that key, rather than for one unlucky request.
- It is per process, so it does not give you one computation globally without additional machinery that has its own failure modes.
- It adds a shared mutable map to the request path, which is a small piece of concurrency-sensitive code you now own.
- Sharing failures avoids retry storms and means a single transient error is amplified to every waiting caller.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe idea — share one in-flight computation among concurrent identical requests — applies to any runtime with a shared address space.
- RUNTIME-SPECIFICOn Node the implementation is a
Map<string, Promise>and it is safe because the check-and-insert can be written without an interveningawait, so nothing interleaves. On a threaded runtime — the JVM, Go, Python threads — the same map needs real synchronisation (computeIfAbsent, a mutex,sync.Once, or Go'ssingleflightpackage), because two threads genuinely execute the check simultaneously. - SCALE-SPECIFICBelow a few concurrent requests per key, coalescing does nothing measurable. Its value scales with the number of arrivals inside one recompute window, which is a function of request rate and computation duration rather than of total traffic.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — why one computation across a fleet requires coordination, and what a lease with a TTL actually guarantees.