The question this answers
The endpoint got faster and the database got slower — where did the load actually go?
An order-history endpoint that fetches 100 orders and, for each, enriches it with line items — changed from a sequential loop to Promise.all over all 100 at once, at 150 requests per second.
The database connection pool (20 connections), the database's own worker slots, the shared buffer cache, and every other service's share of them. None of these appear anywhere in the code that was changed.
Concurrent demand on any shared downstream resource stays below the level at which its service time degrades — the pool never has more waiters than it can drain within the request deadline, and no request holds a connection while waiting for another connection.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The speedup that was never yours to take
On a laptop, against a local database with no other traffic, 100 sequential queries at 3 ms each take 300 ms and 100 parallel queries take about 15 ms. The measurement is real. What it does not include is the other 149 requests per second that will be doing the same thing in production, or the 20-connection pool they all share, or the other endpoints that need that pool too.
The arithmetic that matters is simple and nobody does it: concurrent downstream demand = request rate × fan-out width × downstream latency. At 150 req/s with a fan-out of 100 and a 3 ms query, that is 45 concurrent queries in steady state — against a pool of 20. The pool is now the bottleneck, requests queue for connections, queueing raises latency, higher latency raises concurrency further, and the system finds a new equilibrium far worse than the one it left. This is Little's law being paid attention to only after the incident (Little's Law as Working Intuition).
The curve below is the shape of that. Speedup rises while the downstream has headroom and collapses through it — and note where the peak is: not at the widest fan-out. The best configuration is a bounded one, and the code that shipped had no bound at all.
The interleaving that turns saturation into deadlock
Saturation alone is survivable: requests get slow, some time out, and the system recovers when the burst passes. What is not survivable is the pattern where a request holds a pool connection *while waiting for another pool connection*. Then the pool cannot drain at all, because every holder is waiting on a resource that only a holder can release.
This is a textbook circular wait — the same structure as Deadlock, with the pool as the resource and the fan-out as the second acquisition. It arrives through code that looks nothing like a lock: a transaction opened for the outer query, and a Promise.all inside it for the enrichment.
The fixes are ordered by how much they help. First, never fan out while holding a connection — close the outer query before the inner ones start. Second, bound the fan-out to well under the pool size so the two can coexist. Third, replace the fan-out with one batch query, which removes the problem instead of managing it (Bounding Concurrency, batch-apis).
| # | Request 1 | Request 2 | Request 3 | Request 4 | Connection pool (size 4) | State |
|---|---|---|---|---|---|---|
| 1 | acquires conn A; BEGIN; SELECT orders → 100 rows | · | · | · | · | free=3 held=1 waiting=0 |
| 2 | · | acquires conn B; BEGIN; SELECT orders | · | · | · | free=2 held=2 waiting=0 |
| 3 | · | · | acquires conn C; BEGIN; SELECT orders | · | · | free=1 held=3 waiting=0 |
| 4 | · | · | · | acquires conn D; BEGIN; SELECT orders | · | free=0 held=4 waiting=0 |
| 5 | still holding A, issues Promise.all over 100 enrichment queries — each needs a connection | · | · | · | · | free=0 held=4 waiting=100 |
| 6 | · | same: holds B, queues 100 more acquisitions | · | · | · | free=0 held=4 waiting=200 |
| 7 | · | · | · | · | no connection can be released: every holder is blocked on an acquisition | free=0 held=4 waiting=400 ✕ Circular wait. R1 waits for a connection that only R1 (or R2/R3/R4) can free, and none of them can free one until their own wait completes. The pool is deadlocked with four connections and four hundred waiters. |
| 8 | · | · | · | · | acquisition timeouts fire at 30 s; every request fails; clients retry | free=0 held=4 waiting=800 ✕ Retries double the waiter count against a pool that is still deadlocked. The database is idle — it is doing no work at all — and the service is completely down. |
What actually changed, measured on both sides
The reason this ships is that the pull request is a two-line diff with a benchmark attached, and the benchmark is honest about the only thing it measured. The table below is what the same change looks like when you measure both sides of the boundary — and it is the table to ask for before approving the change.
The remedies are not exotic. A concurrency bound is a few lines. A batch query is usually already supported. What both require is knowing the downstream pool size, and that number is typically owned by nobody on the team making the change. That gap is the real cause of this incident class, not the Promise.all.
| Signal | Sequential (before) | Unbounded fan-out (after) | Bounded to 8 (fixed) | Batch query (best) |
|---|---|---|---|---|
| Endpoint p50, laptop | 300 ms | 15 ms | 45 ms | 8 ms |
| Endpoint p99, production | 340 ms | 30,000 ms (timeout) | 90 ms | 14 ms |
| Concurrent DB queries at 150 req/s | ~45 | ~4,500 offered, pool-capped at 20 | ~360 offered, pool-capped at 20 | ~2 |
| Pool wait time p99 | 2 ms | 30,000 ms (acquisition timeout) | 35 ms | 0 ms |
| Queries per request | 101 | 101 | 101 | 2 |
| DB CPU | 35% | 8% — it is idle and deadlocked | 48% | 11% |
| Blast radius | This endpoint | Every endpoint sharing the pool | This endpoint | This endpoint |
| What the trace blames | — | The database | — | — |
Key points
- Parallelising a fan-out does not reduce work; it concentrates the same work into a shorter window, and concentration is what saturates shared resources.
- Concurrent downstream demand equals request rate times fan-out width times downstream latency — do this arithmetic before the deploy, not during the incident.
- The speedup curve turns *down* past the knee rather than flattening, because queueing causes timeouts and timeouts cause retries that add more load.
- Fanning out while holding a pool connection is a circular wait: the pool deadlocks while the database sits idle, and every database dashboard says healthy.
- The blast radius is every endpoint sharing that pool, not the endpoint that was changed.
- A bound chosen against the downstream pool size gets most of the speedup with none of the collapse.
- One batch query beats any amount of well-tuned concurrency, and it usually already exists.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A sequential loop offers one unit of concurrent demand downstream per in-flight request, for the duration of the whole loop.
- • A parallel fan-out offers N units per in-flight request, for a much shorter duration — the same total work, N times the peak.
- • Downstream, that peak meets a finite resource: a connection pool, a worker count, a rate limit, a thread pool.
- • While demand is below capacity, service time is flat and the speedup is close to linear.
- • Above capacity, requests queue; queue wait adds to latency; higher latency means each request holds its resources longer; concurrency rises further. The feedback loop is positive.
- • At the timeout threshold, requests fail and clients retry, adding fresh demand to an already saturated resource — the curve's downward turn.
- • If any holder of the resource waits for another unit of the same resource, the queue cannot drain at all and the pool deadlocks independently of the downstream service's health.
- • R1 through R4 each hold a pool connection with an open transaction and each queue 100 more acquisitions; no connection can be released; the pool deadlocks with four connections held and four hundred waiters, while the database does no work.
- • At 150 req/s with fan-out 100 and 3 ms queries: 45 queries in flight in steady state against a pool of 20, so 25 wait; the wait pushes per-request latency up, which raises in-flight count, which lengthens the wait.
- • Acquisition timeouts fire at 30 s; every waiting request fails; clients retry; the waiter count doubles against the same saturated pool.
- • A second, unrelated endpoint that needs one connection per request now waits behind 400 enrichment acquisitions and times out — the endpoint that was never changed is the one that pages someone.
- • Bounded to 8: at most 8 enrichment queries per request are outstanding, so offered concurrency is ~360 rather than ~4,500; the pool stays under 90% and p99 is 90 ms.
- • Batch query: two queries per request instead of 101, concurrent demand of ~2, and the fan-out problem no longer exists to be tuned.
- • Guaranteed: total downstream work is unchanged by parallelising. The same queries run, over the same rows.
- • Guaranteed: peak concurrent demand multiplies by the fan-out width.
- • Guaranteed: a bound on the fan-out caps that multiplication at the bound, regardless of input size.
- • NOT guaranteed: that the downstream resource can absorb it. Nothing in the calling code knows the pool size.
- • NOT guaranteed: that the speedup survives contact with production traffic. The laptop benchmark measured an uncontended system.
- • NOT guaranteed: that the failure appears in the changed service. It appears in whoever shares the pool, and the traces blame the database.
- • NOT guaranteed: recovery. Once retries are compounding, the system may not return to baseline without shedding load (Retry Storms: The Load You Generated Yourself).
- • The connection pool is the first ceiling and usually the one that breaks; its wait time is the signal that matters and is rarely on a dashboard.
- • The database's own worker slots and buffer cache are the second: 4,500 offered concurrent queries would thrash the cache even with an unlimited pool.
- • Third-party rate limits are the third, and they fail differently — as 429s attributed to your service, sometimes with a penalty window.
- • The application's own event loop or thread pool is the fourth: N times the concurrent continuations means N times the resumption work in the same tick.
- • Every other consumer of the shared resource is contending too, which is why the blast radius is the pool, not the endpoint.
- • Connection-pool exhaustion: acquisition waits grow past the request timeout and unrelated endpoints fail first.
- • Pool deadlock: holders waiting for more of the resource they hold, so the queue cannot drain and the database is idle throughout.
- • Retry amplification: timeouts produce retries that add demand to a saturated resource, preventing recovery (Retry Storms: The Load You Generated Yourself).
- • Rate-limit rejection from third parties, sometimes with a lockout longer than the burst that caused it.
- • Misattribution: traces and dashboards blame the database, which is healthy, so the investigation starts in the wrong place.
- • Cross-service blast radius: the endpoint that fails is not the endpoint that changed.
- • Silent regression: at low traffic the change is a pure win, so it passes staging and fails at peak.
- • When the fan-out width is small and fixed — four calls on a product page, not one hundred derived from the input.
- • When the calls go to *different* downstream systems, so no single resource absorbs the multiplication.
- • When the downstream has verified headroom at your peak request rate, and you have the pool-utilisation graph to prove it.
- • When the fan-out is bounded to a width chosen against the downstream capacity rather than against the input length.
- • When the width is derived from data — a list, a page size, a search result count — because it is then unbounded by construction.
- • When every call targets the same pool, the same table or the same rate limit.
- • When the fan-out happens inside an open transaction or while otherwise holding a unit of the contended resource.
- • When the downstream is shared with other services whose owners were not part of the change.
- • When the request rate is high, because the multiplier applies at peak, which is when headroom is smallest.
- • Before: pool utilisation and acquisition wait time at p99, at production peak. Without this baseline the after-numbers mean nothing.
- • Downstream queries per inbound request, and concurrent queries in flight — the two numbers that make the multiplication visible.
- • Pool wait time as a distribution. It is near zero until it is catastrophic; the mean tells you nothing.
- • Ratio of downstream request rate to inbound request rate. It should equal the fan-out width; anything higher is retries compounding.
- • Database-side concurrency and CPU. A saturated pool with an idle database is the deadlock signature and is unmistakable once you look for it.
- • Error rate on endpoints that share the pool but were not changed — the earliest signal that the blast radius has escaped.
- • Load-test the fan-out at production request rate, not with one request; a single request will never reproduce this (Load Test Shapes: The Shape Is the Hypothesis and
load-testing).
- • You now own a concurrency bound that must be justified against a resource owned by another team, and re-justified whenever either side changes.
- • Bounded mapping needs an implementation — a semaphore, a queue, or a library — plus a decision about what happens when the bound is reached: wait, or reject.
- • The relationship between application concurrency and downstream capacity has to be documented somewhere, or it will be rediscovered by the next incident.
- • Load tests must now model the fan-out at realistic request rates, which is a materially more expensive test than the one that existed.
- • One batch query —
WHERE id IN (...), a batch endpoint, a join. Removes the fan-out rather than tuning it, and it is almost always available (batch-apis, The Comb: N+1 as a Visible Shape). - • Bounded concurrency with a semaphore or a
mapWithConcurrency(items, K, fn)helper, with K chosen against the downstream pool (Semaphores: Counting Permits as a Resource Limit, Bounding Concurrency). - • Keep it sequential. If the endpoint is cold or the total is acceptable, sequential is the safest thing in this lesson and costs nothing to operate.
- • Move the work off the request path: precompute, cache, or materialise the enriched view so no fan-out is needed at read time.
- • A dedicated pool for the fan-out, so saturation cannot reach the endpoints that share the main one — bulkheading, which limits the blast radius without fixing the cause.
- • Backpressure at the edge: cap in-flight requests per instance so the fan-out multiplier applies to a bounded base (Backpressure,
backpressure).
One request, N downstream calls
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
The producer is faster than the consumer
Why is 8 cores only 4.5×?
| workers | ideal | Amdahl only | realistic | limited by |
|---|---|---|---|---|
| 1 | 1.0× | 1.00× | 1.00× | none |
| 2 | 2.0× | 1.90× | 1.85× | serial |
| 4 | 4.0× | 3.48× | 3.19× | serial |
| 6 | 6.0× | 4.80× | 4.17× | serial |
| 8 | 8.0× | 5.93× | 4.17× | bandwidth |
| 10 | 10.0× | 6.90× | 4.17× | bandwidth |
| 12 | 12.0× | 7.74× | 4.17× | bandwidth |
| 14 | 14.0× | 8.48× | 4.17× | bandwidth |
| 16 | 16.0× | 9.14× | 4.17× | bandwidth |
What people believe, and what is true
It is the same amount of work, so it cannot hurt the database.
Same work, N times the peak concurrency. Shared resources are sized for peak concurrency, not for total work, and that is exactly the quantity this change multiplies.
The database fell over, so we need a bigger database.
Check whether the database was doing anything. A saturated pool with an idle database is an application-side deadlock, and a bigger database fixes none of it.
It passed staging.
Staging has a fraction of the request rate. The multiplier is request rate times fan-out width; at one-fiftieth of the traffic the change is a pure win, which is exactly why it shipped.
Go deeper
Overview
Parallelising a fan-out concentrates the same work into a shorter window. Shared downstream resources are sized for concurrency, and that is what you just multiplied.
Practical
Compute request rate times fan-out width times downstream latency and compare it with the pool size. Bound the fan-out, never fan out while holding a connection, and prefer one batch query.
Advanced
Past saturation the feedback is positive: queueing raises latency, latency raises concurrency, timeouts produce retries. The system does not degrade gracefully and does not recover on its own — it needs shedding or a bound.
Internals
Little's law relates in-flight count, arrival rate and service time; the fan-out multiplies arrival rate at the downstream resource while its service time is fixed, so in-flight count rises linearly until the queue forms and service time itself begins to degrade.