The question this answers
What number limits how many of these run at once — and is that number one I chose?
A nightly reconciliation job that loads 12,400 order ids and calls await Promise.all(ids.map(reconcile)), where each reconcile makes two HTTP calls and one database query.
Nothing in application memory — and that is exactly why this is missed. What is shared is the process's file descriptor table, its socket buffers, the connection pool, and the downstream services' capacity, none of which appear in the code.
The number of operations in flight at any moment stays below the smallest ceiling in the chain — descriptors, pool size, downstream capacity and rate limit — regardless of how large the input is.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The input size is not a limit
The whole defect is one substitution: the code says "run all of these concurrently" and the author reads "run a reasonable number concurrently", because when they wrote it the array had eleven elements. Nothing in the code changes when the array has twelve thousand. The concurrency limit was never chosen; it was inherited from whatever the data happens to contain, which is a number that grows with your business.
The thread version is the same defect with a heavier unit. A thread per request works beautifully up to some hundreds; each thread costs a stack (commonly measured in megabytes of reserved address space), a kernel structure and a slot in the scheduler. At ten thousand, memory and context-switching dominate and throughput falls as concurrency rises — which is Oversubscription, and it is why thread pools exist at all.
What makes this so persistent is that neither version produces a warning. There is no error at 100, no error at 1,000, and at some load-dependent point the process hits a ceiling that belongs to someone else — the operating system, the connection pool, the downstream API — and fails in that system's vocabulary rather than in yours.
1// 1. Concurrency = input length.2await Promise.all(ids.map(reconcile)) // 12,400 in flight3 4// 2. Concurrency = arrival rate, with no ceiling.5server.on('request', (req) => { new Thread(() => handle(req)).start() })6 7// 3. Concurrency = whatever recursion produces.8async function crawl(url: string) {9 const links = await fetchLinks(url)10 await Promise.all(links.map(crawl)) // branching factor ^ depth11}12 13// The fix is the same shape every time: a number you chose, enforced.14const limit = pLimit(24) // or a semaphore, or a pool15await Promise.all(ids.map((id) => limit(() => reconcile(id))))16 17// And the number comes from the tightest downstream ceiling, not from a guess:18// pool size 20 | partner API 50 rps | ulimit -n 102419// -> 24 in flight keeps every one of those below its limit, with headroom.Which ceiling you hit first
The failure is rarely memory, which is why "we have plenty of RAM" is such a misleading defence. The ceilings arrive in a fairly predictable order, and each presents in a different system with a different error message — which is the main reason these incidents take so long to diagnose.
File descriptors usually come first: every socket is a descriptor, the per-process limit is often 1024 by default, and the failure is EMFILE or ENFILE — which will also break unrelated things, like opening a log file or accepting a connection. Then the connection pool, which does not error but blocks, so the symptom is timeouts rather than refusals. Then the downstream rate limit, which returns 429s that a naive retry loop converts into more load. Then downstream saturation, where you are not rate-limited but you are the reason that service's p99 went to eight seconds. Memory is often last, and by then you have already caused three other incidents.
The most damaging property is that the blast radius extends beyond your process. Twelve thousand concurrent requests to a partner API is a denial-of-service attack you are performing accidentally, and the partner will describe it that way. Bounding concurrency is therefore not only a stability measure for your own service; it is the contract you keep with everything downstream. See Bounding Concurrency and Backpressure.
| Ceiling | Typical value | How it presents | Why it is confusing |
|---|---|---|---|
| File descriptors per process | 1024 default, often unraised | EMFILE / ENFILE, and unrelated file opens start failing | The error names sockets, but your log writer breaks too |
| Ephemeral ports / socket buffers | Tens of thousands, per destination tuple | Connection failures and TIME_WAIT accumulation | Looks like a network problem in someone else's system |
| Connection pool | 10-50 typical | Timeouts waiting for a connection, not errors | Presents as database slowness with the database idle |
| Downstream rate limit | 50-1000 rps, contractual | 429s, sometimes an account-level block | A naive retry loop turns the limit into an outage |
| Downstream capacity | Unstated, discovered | Their p99 rises; your p99 rises with it | You caused an incident in a service you do not own |
| Thread stacks / memory | MBs per thread, GBs per process | OOM kill or allocation failure | Usually the last ceiling, so "we have RAM" proves nothing |
| Scheduler | Cores, not threads | Context switches dominate; throughput falls as concurrency rises | CPU looks busy while useful work decreases |
Throughput does not keep rising, and then it falls
The intuition that makes unbounded concurrency feel safe is that more in flight means more throughput. That holds up to the point where some resource saturates, then flattens, then reverses — because past saturation the extra concurrency adds queueing, context switching, cache pressure and retry traffic without adding capacity.
For an I/O-bound workload the useful ceiling is set by the downstream, not by your cores. Twenty-four concurrent calls against a service that can handle fifty requests per second is close to optimal; twelve thousand does not make that service faster, it makes every one of your requests wait in its queue and then time out. The work you are doing at that point is mostly queueing, and the timeouts convert it into work you throw away — which is where the curve turns over rather than flattening. See Queueing: Why Systems Get Slow Before They Get Broken in Observability & Performance for the underlying model.
The practical consequence: the right bound is discovered by measurement against the constrained resource, not derived from a formula, and it is a property of the *downstream*, not of your machine. There is no universal number, and anyone who offers one has not asked what you are calling. Set it, enforce it, and expose it as configuration so it can be changed during an incident without a deploy.
Key points
- A concurrency level derived from the input size is not a bound — it is whatever your data grows into.
- The first ceiling is usually file descriptors or the connection pool, not memory, and each presents in a different system's vocabulary.
- Unbounded fan-out exports the failure: twelve thousand concurrent calls is an accidental denial of service against whoever you are calling.
- Throughput does not plateau past saturation, it falls, because the excess concurrency produces timeouts and retries rather than results.
- The right bound comes from the tightest downstream ceiling, is found by measurement, and belongs in configuration so it can change without a deploy.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Find every place concurrency is created: a map over an array into Promise.all, a spawn per request, a recursive fan-out, an unbounded queue consumer.
- • For each, write down the number that limits it. If the answer names the input, the arrival rate, or the branching factor, it is unbounded.
- • Identify the tightest ceiling in the chain — descriptors, pool size, rate limit, downstream capacity — and set the bound below it with headroom.
- • Enforce it with a semaphore, a pool, or a concurrency-limited map; the mechanism matters far less than the number existing at all.
- • Decide what happens when the bound is reached: block the producer, reject with a clear error, or shed load. "Queue without limit" is not one of the options.
- • Expose the number as configuration and emit in-flight count as a metric, so the bound is observable and adjustable during an incident.
- • Unbounded fan-out: 12,400 reconcile calls start within milliseconds; each opens a socket; at descriptor 1,024 every subsequent open fails with EMFILE, including the log file, and the error handler cannot write.
- • Pool starvation: 12,400 tasks each await a connection from a pool of 20; 20 proceed, 12,380 wait; the pool's acquire timeout fires and the job reports 12,380 database timeouts against an idle database.
- • Rate-limit cascade: the partner API returns 429 to 11,000 of 12,400 calls; a naive retry retries all of them; the retry traffic exceeds the original traffic and the account is blocked.
- • Thread explosion: 10,000 concurrent requests each spawn a thread; the scheduler run queue is 10,000 deep on 8 cores; context switches dominate, per-request latency rises 50x and throughput falls below the 200-thread case.
- • Recursive crawl: depth 4 with a branching factor of 30 produces 810,000 concurrent fetches from three lines of code that contain no loop.
- • Bounded: a semaphore of 24 admits 24; task 25 waits on acquire; in-flight never exceeds 24 regardless of whether the input has 100 or 100,000 elements, and the job completes.
- • Promise.all guarantees every promise is awaited and that the result array preserves input order; it guarantees NOTHING about how many run concurrently.
- • It does NOT cancel the remaining promises when one rejects — they keep running, unobserved, and their failures may surface as unhandled rejections.
- • A thread-per-request server guarantees isolation between requests; it does NOT guarantee the machine can support the arrival rate.
- • A semaphore guarantees at most N permits are outstanding; it does NOT guarantee a permit released on every error path unless you wrote it that way.
- • A bound on your side does NOT guarantee the downstream is protected if several of your instances each apply it independently — N instances times the per-instance bound is the real number.
- • Nothing about being I/O-bound guarantees concurrency is free; each in-flight operation holds a descriptor, a buffer and usually a pool slot.
- • Contention appears in resources the code never mentions: descriptor tables, socket buffers, the connection pool and the downstream's own queues.
- • Past the downstream's capacity, additional concurrency converts directly into queue time at the downstream, which becomes timeouts on your side.
- • Retry traffic after timeouts is self-amplifying and is often larger than the original load, which is what prevents recovery. See Thundering Herd.
- • With threads, the scheduler and the cache become the contended resources: run-queue depth rises, cache warmth is destroyed, and per-thread throughput falls.
- • Across a fleet, the effective concurrency against a shared downstream is per-instance concurrency times the instance count, which is the number most teams forget to compute.
- • File descriptor exhaustion (EMFILE/ENFILE), which breaks unrelated operations in the same process.
- • Connection pool exhaustion presenting as database timeouts while the database is idle.
- • Downstream rate limiting, escalating to account-level blocking under naive retries.
- • Downstream saturation — an incident in a service you do not own, caused by your job.
- • Memory exhaustion or OOM kill, typically after several of the above have already fired.
- • Oversubscription with threads: throughput falling as concurrency rises, with high CPU and low useful work.
- • Unhandled rejections from tasks still running after Promise.all rejected.
- • Stack overflow or memory exhaustion from recursive fan-out with no depth or width limit.
- • Unbounded concurrency is genuinely fine when the input is small and fixed by construction — three parallel calls to build one response page.
- • It is fine when the operations are pure computation with no external resource, and the runtime already bounds parallelism at the core count.
- • Thread-per-request is a legitimate model at bounded, well-understood concurrency, and its simplicity is worth real money. See Thread per Request: The Model That Reads Like Ordinary Code.
- • The unbounded form is a reasonable first draft — provided the bound is added before the input can grow, and "before" means before it ships.
- • Whenever the collection size is data-dependent, which is almost always, and especially when it is customer-dependent.
- • Whenever each unit touches an external resource: a socket, a connection, a partner API, a file.
- • Whenever the fan-out is recursive, where a small branching factor becomes an enormous width in four levels.
- • Whenever the process runs on many instances, since each one applies its own bound and the downstream sees the sum.
- • Whenever retries are involved, because the unbounded design and the retry loop amplify each other.
- • In-flight operation count as a live gauge, per downstream. If you cannot answer "how many are running right now", the bound does not exist operationally even if it exists in code.
- • Open file descriptors against the process limit, which is the earliest and most specific leading indicator.
- • Connection pool wait time and acquisition timeouts, which detect pool starvation before it becomes a database story.
- • Downstream 429 rate and downstream p99, which tell you whether your bound is a good neighbour or merely a survival measure.
- • Throughput against concurrency, sampled at increasing bounds — the point where it stops rising is the number you want, and the point where it falls is where you currently are.
- • Fleet-wide concurrency against a shared downstream: per-instance bound times instance count, tracked as one number.
- • You now own a number, which means owning where it comes from, how it is configured, and who changes it during an incident.
- • You need a policy for exceeding it — block, reject or shed — and each choice propagates a different behaviour upstream.
- • Permit release must be correct on every error path, or the bound erodes silently until concurrency is zero.
- • Fleet-level coordination is a further step: per-instance bounds do not compose into a global one without either a shared limiter or arithmetic nobody keeps up to date.
- • Bounded concurrency makes the job take longer, which is a visible regression somebody will ask you to revert.
- • A worker pool or a concurrency-limited map, which is the same fix with the bound built into the structure. See Thread Pools and Bounding Concurrency.
- • Batching: one request carrying 500 ids instead of 500 requests, which frequently removes the concurrency question entirely.
- • A queue with a fixed number of consumers, which converts a burst into a controlled rate and gives you durability as a bonus.
- • Streaming with backpressure, so the producer is slowed by the consumer rather than buffering ahead of it. See Backpressure.
- • Doing it sequentially. For a nightly job, 12,400 sequential operations at 40 ms each is eight minutes, which is very often entirely acceptable and has no failure modes at all.
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
One of five fails — what happens to the siblings?
Promise.all rejects on the FIRST rejection; siblings are NOT cancelled and keep running
Promise.allSettled never rejects; resolves at 90 ms with {status, value|reason} for all five
asyncio.gather(...) return_exceptions=False → raises at 40 ms, siblings still NOT cancelled
return_exceptions=True → returns at 90 ms with the exception as a value
asyncio.TaskGroup the structured alternative: on failure it CANCELS the siblings, then raisesOne request, N downstream calls
More workers than cores
What people believe, and what is true
It is I/O-bound, so concurrency is free.
Free in CPU, not in descriptors, sockets, pool slots or downstream capacity. The first ceiling you hit is usually the descriptor table, and it breaks unrelated operations in the same process.
The array is finite, so the concurrency is bounded.
Bounded by the data, which is not a bound you chose and grows with the business. A bound is a number derived from what the constrained resource can absorb.
More concurrency means more throughput.
Up to the saturation point of the tightest resource. Past it, throughput flattens and then falls, because the extra concurrency produces queueing, timeouts and retries rather than completed work.
We set a limit of 50 per instance, so the partner sees 50.
The partner sees 50 times your instance count. Per-instance bounds do not compose; either compute the fleet number deliberately or use a shared limiter.
Go deeper
Overview
If the number of things running at once comes from the length of a list or the rate of arrivals, there is no limit. Pick a number, enforce it with a semaphore or a pool, and make it configurable.
Practical
Derive the number from the tightest ceiling in the chain — pool size, rate limit, descriptor limit — with headroom. Emit in-flight count as a metric. Decide what happens when the bound is reached and write it down.
Advanced
Measure throughput against increasing bounds and take the point where the curve stops rising. Then compute the fleet-wide number, because per-instance limits multiply by instance count at the shared downstream.
Internals
Every in-flight network operation costs a descriptor, kernel send and receive buffers, and a slot in the connection pool. Descriptors are a per-process table with a hard limit; exhausting it fails every subsequent open in the process, including the ones your error handling depends on. That coupling is why this failure looks like several unrelated failures at once.