Unbounded Concurrency
Work that starts without a limit does not fail gracefully — it consumes memory, connections and downstream capacity until something breaks.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What stops my service starting more work than it can finish?
An admin exports a report that needs details for 5,000 customers from a partner API. It should be as fast as possible.
Map over the ids and fire all the requests at once — await Promise.all(ids.map(fetchCustomer)). The runtime handles concurrency, and it is much faster than a loop.
5,000 simultaneous HTTP requests open 5,000 sockets, allocate 5,000 response buffers, and hold 5,000 promise contexts. Memory climbs until the process is killed by the platform (Memory Leaks in Backend Services).
- 5,000 simultaneous HTTP requests open 5,000 sockets, allocate 5,000 response buffers, and hold 5,000 promise contexts. Memory climbs until the process is killed by the platform (Memory Leaks in Backend Services).
- The partner API rate-limits at a fraction of that, so most requests come back 429 and the "fast" version is slower than a bounded one — plus you may be throttled for the rest of the hour (Rate Limiting).
- The same pattern against your own database opens more queries than the pool has connections; every request in the service — including unrelated ones — now waits for a connection (Connection Pool Exhaustion).
- One slow response holds the whole batch open, because
Promise.allresolves when the last one does, and every buffer stays allocated until then (Promise.all & gather). - The failure is not graceful. It is an OOM kill or a pool timeout, and the request that triggered it usually was not the one that failed.
What is actually happening
- Starting work is cheap; finishing it is not. Every in-flight operation holds resources — a socket, a buffer, a connection, a stack or a continuation — and those resources are finite in ways the code that starts the work does not see (Bounding Concurrency).
Promise.all,asyncio.gather, an unbounded goroutine loop and a thread-per-item pattern all express "start everything". None expresses "start at most N".- Without a bound, the limit is discovered rather than chosen: it is whichever finite resource runs out first, and its failure mode is whatever that resource does when exhausted — an allocation failure, a connection timeout, a 429, a kernel refusing sockets.
- The pool is the bound people forget they have. A connection pool of 20 silently limits concurrency to 20, but only after every waiter has already been created and is queued, holding whatever it holds (Connection Pools).
- Bounding converts unbounded resource growth into bounded resource use plus queueing. That is a strictly better failure mode, because a queue can be measured, shed and given a timeout (Backpressure).
- The bound belongs where the work is started, not only at the edge. Rate limiting inbound requests does not bound the fan-out one request can create (Fan-Out: Waiting for the Slowest of Seven).
The limit you did not choose is the one you get
asyncio.Semaphore with async with, which handles release automatically; Go uses a buffered channel as a counting semaphore or errgroup.SetLimit. The failure the pattern prevents is identical in all three.Code that starts unbounded work still has a limit. It is just not in the code — it is whichever finite resource is exhausted first, discovered at runtime, in production, under load.
That matters because the discovered limit has a bad failure mode by construction. Memory exhaustion kills the process. Pool exhaustion fails unrelated requests. A downstream rate limit may block you for a fixed window. None of these degrade proportionally, and none of them are visible until they happen.
A chosen limit converts all of that into queueing. Work waits instead of failing, the wait is measurable, and you can decide what to do when the queue grows — shed, reject, or scale. That is the entire argument for bounding, and it is why the number matters less than its existence.
const results = await Promise.all( ids.map(id => fetchCustomer(id)) // 5,000 sockets, at once ) // Memory holds 5,000 in-flight requests and every response buffer // until the slowest one returns. One rejection discards all of it.
const LIMIT = 10 // from the partner's rate limit
const sem = new Semaphore(LIMIT)
const settled = await Promise.allSettled(
ids.map(async (id) => {
await sem.acquire()
try {
return await withTimeout(fetchCustomer(id), 5_000)
} finally {
sem.release() // released on every path
}
})
)
inFlight.set(0)
const ok = settled.filter(r => r.status === 'fulfilled')
metrics.increment('export.failed_items', settled.length - ok.length)The bounded version holds at most ten sockets and ten response buffers at any moment, so memory is a function of the limit rather than of the input size. The timeout stops one stuck call from occupying a permit indefinitely, finally guarantees the permit returns even on failure, and allSettled keeps 4,990 successes when ten fail. The unbounded version is faster only if the partner can absorb 5,000 concurrent requests, and it cannot.
Where the resource actually runs out
Diagnosing an unbounded-concurrency incident is a matter of identifying which resource was exhausted, because the symptom rarely points at the code that caused it. An OOM kill names no endpoint. A pool timeout appears on whichever request happened to ask for a connection next.
The table below is the mapping worth knowing. Note how often the visible failure is somewhere other than the fan-out: that displacement is what makes these incidents hard to attribute, and it is why the in-flight gauge is worth adding before you need it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| 5,000 concurrent HTTP calls | Process OOM-killed; all in-flight requests lost | Response buffers and continuations held simultaneously | Semaphore-bounded fan-out; stream instead of accumulate (Bounding Concurrency) |
| Unbounded queries against a 20-connection pool | Unrelated endpoints time out acquiring connections | Fan-out waiters occupy the whole pool | Bound below pool size; separate pool for batch work (Bulkheads) |
| Fan-out to a rate-limited partner | Mass 429s, then a temporary block | Burst exceeds their limit | Limit to their documented rate; respect Retry-After (Rate Limiting) |
| Per-request bound, many concurrent requests | Aggregate concurrency far above the bound | The limit is per request, not global | A shared limiter, or a worker pool sized globally (Worker Scaling) |
| Retries inside a bounded fan-out | Effective concurrency multiplied by attempt count | The limiter sits inside the retry rather than outside | Acquire the permit outside the retry loop (Retries) |
| Unbounded rows loaded into memory | Memory proportional to result size; OOM on the biggest tenant | No pagination or streaming | Cursor-based iteration (Pagination That Survives a Large Table) |
| Thread-per-item on a thread-pool runtime | Thread exhaustion; latency collapse before memory | Stacks are megabytes each | Fixed-size pool sized to cores and workload (Sizing a Thread Pool) |
Choosing the number
The right limit comes from the tightest downstream constraint, not from a round number. Work outward from what the fan-out touches: if the partner permits 10 requests per second and each takes about a second, roughly 10 in flight saturates them and more only produces 429s. If each response is a megabyte and you can afford 100 MB for this path, 100 is your ceiling regardless of anything else.
Where several constraints apply, take the minimum, then leave headroom for the fact that this is not the only thing running. A fan-out sized to the entire connection pool starves every other request in the process.
And then measure. In-flight count, wait time and downstream rejection rate together tell you whether the number is right: consistently at the limit with rising wait time means the bound is the bottleneck; never near it means the bound is not the constraint and something else is.
| Constraint | How to derive the number | What it looks like when wrong |
|---|---|---|
| Downstream rate limit | Permitted rate x average duration | Sustained 429s; possible temporary block |
| Connection pool | Pool size minus headroom for normal traffic | Unrelated endpoints timing out on acquire |
| Memory per operation | Affordable bytes / bytes per in-flight response | OOM kill correlated with large inputs |
| CPU per result | Cores, for CPU-bound post-processing | Event loop lag or thread starvation (Blocking the Event Loop) |
| Downstream service capacity | Their stated concurrency budget for you | You cause their incident, then they cause yours |
| Instance count | Global budget / instances, if the limiter is per instance | Aggregate concurrency scales with your deploy size |
| Nothing measurable yet | Pick a small number, instrument, adjust | Better than unbounded on the first day it matters |
How to build it
Most important first.
- Bound every fan-out with an explicit concurrency limit — a semaphore, a worker pool, a chunked loop, a library limiter. The number should be visible in the code and configurable (Semaphores: Counting Permits as a Resource Limit).
- Choose the limit from the tightest downstream constraint: the partner's rate limit, your pool size minus headroom for other traffic, the memory a response buffer costs times N.
- Give every in-flight operation a timeout, so a bounded set cannot be held open indefinitely by one stuck call (Timeouts).
- Prefer streaming over batching where the result set is large — process and release as you go rather than accumulating everything in memory (Pagination That Survives a Large Table).
- Use
allSettled-style semantics for partial failure, so one error does not discard 4,999 successful results. - Isolate fan-out from user-facing traffic: a separate pool, worker set or bulkhead, so an export cannot starve the API (Bulkheads).
- Move large fan-outs out of the request path entirely. A 5,000-item export is a job, not a request (Request or Background?).
What can go wrong
- OOM kill of the whole process, taking every unrelated in-flight request with it.
- Pool exhaustion, where the fan-out's waiters occupy every connection and the rest of the service returns errors for reasons unrelated to its own load (Cascading Failure).
- Downstream rate limiting or, worse, being blocked by a partner who sees your burst as abuse.
- A bound applied per request but not globally, so ten concurrent requests each running a bounded fan-out of 20 still produce 200 concurrent calls.
- A semaphore acquired and not released on the error path, so the limit tightens with every failure until nothing can proceed (Semaphore versus Mutex: Not the Same Primitive).
- Retries multiplying concurrency: each of N in-flight calls retrying three times is up to 3N in flight (Retry Storms).
- Bounded concurrency with no timeout, so the bound holds and the work never completes.
- Concurrent operations in a fan-out writing shared state — an accumulator array, a progress counter — safely only if the runtime's model guarantees it (Shared Mutable State).
- A semaphore released twice on a path that both catches and finally-releases, silently raising the limit.
- A global limiter shared across instances with no coordination, so the effective limit is the configured limit times the instance count.
- Retries and the original both counted against the bound, or neither, depending on where the limiter sits relative to the retry (Retries).
- Unbounded fan-out driven by user input is an amplification primitive: a request containing 100,000 ids becomes 100,000 outbound calls. Validate and cap collection sizes at the edge (Transport Validation).
- It is also a self-inflicted denial of service — the cheapest way for one caller to take down the service for everyone (Resource Limits).
- If the fan-out targets user-supplied URLs or hosts, unbounded concurrency turns an SSRF into a scanner (SSRF — When the Backend Fetches a URL).
- "
Promise.allis parallel, so it is faster." It starts everything at once. Whether that is faster depends entirely on whether the bottleneck is downstream, and it usually is (Promise.all & gather). - "The pool limits concurrency, so I am safe." The pool limits *execution*. Every waiter still exists, holding memory and a queued request (Connection Pools).
- "Async means non-blocking means free." Each pending operation holds a continuation and its captured state. Ten thousand of them is ten thousand allocations (Async Is Not Parallelism).
- "We rate limit, so we are protected." Inbound rate limiting bounds requests, not the work each request creates.
- "It works in staging." Staging has smaller datasets. Fan-out scales with data size, and that is the variable staging changes most.
Operating it
- In-flight operation count as a gauge, per fan-out site. If it is not measured, the bound is not verifiable.
- Semaphore or pool wait time, which is where bounded concurrency shows up as latency instead of as failure (Concurrency Limits: An Unbounded Server Is a Slower Server).
- Process memory alongside in-flight count — the correlation is what identifies unbounded fan-out as the cause of an OOM.
- Downstream 429 counts and pool acquire timeouts, which are the two most common first symptoms.
- Event loop lag or thread pool saturation, depending on the runtime, since both spike when too much is in flight (Blocking the Event Loop).
- The problem is not proportional to traffic; it is proportional to fan-out per request times concurrent requests. A rare endpoint with a large fan-out is more dangerous than a busy one without.
- At 10x request rate, per-request bounds are no longer sufficient — you need a global bound, because the per-request limit multiplies by the number of concurrent requests.
- At 100x, fan-out belongs in a separate service or worker fleet with its own capacity, so its resource envelope is independent of the API's (Bulkheads).
- Bounded systems degrade by queueing, which is measurable and sheddable. Unbounded systems degrade by dying, which is neither.
- A bound makes the best case slower and the worst case survivable. That trade is almost always right in a service and almost always wrong in a one-off script, which is where the habit of unbounded fan-out comes from.
- Choosing the limit is genuinely hard and context-dependent, and a wrong-but-present limit is far better than none.
- Global bounds are more correct than per-request bounds and need shared state, which on multiple instances means either a distributed limiter or a per-instance limit divided by instance count (A Mutex on Server A Does Nothing About Server B).
- Streaming avoids the memory problem and complicates error handling, because partial output may already have been sent.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALEvery runtime can start more work than it can finish; only the resource that runs out first differs.
- RUNTIME-SPECIFICNode holds a continuation and a socket per pending promise on one loop thread, so the ceiling is memory and file descriptors and the symptom is an OOM kill. CPython with asyncio behaves similarly; with threads, the ceiling arrives sooner because each thread has a stack measured in megabytes. Go goroutines are cheap enough that a million can be started, which moves the failure from the goroutine cost to whatever they all contend on — usually the connection pool or the remote service.
- SCALE-SPECIFICUnbounded fan-out over 10 items is fine and always will be. The danger is that the collection size is usually driven by data, so the code that is correct today becomes the outage when a customer arrives with 50,000 rows.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — load shapes that expose unbounded fan-out: large inputs rather than high request rates.