The question this answers
I have ten thousand independent tasks and firing them all at once destroys something downstream — where exactly do I put the limit, and how do I pick it?
Ten thousand product records to enrich, each requiring one call to a partner API that permits 50 concurrent connections and returns 429 above that.
The permit count — a single integer every task decrements to enter and increments to leave — and the bounded downstream resource the permits stand for. The permit count is a *model* of that resource, and the two drifting apart is the failure mode.
At most N tasks are inside the guarded region at any instant, and every acquired permit is released exactly once — including on the exception path, the cancellation path and the early-return path.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The pattern, and the three lines that are actually load-bearing
The mechanism is a counting semaphore: initialise it with N permits, have each task acquire before entering the expensive region and release after leaving. Operating Systems covers what a semaphore *is* and how the kernel implements the wait — see Semaphores: Counting Permits as a Resource Limit for the permit-counting model and OS's semaphores lesson for the primitive. This lesson is about using it as a deliberate design element rather than a rescue after an incident.
What matters in the code below is not the acquire. It is (a) that the release is in a finally, so a task that throws still gives its permit back; (b) that the *guarded region is only the expensive part*, not the whole task, because a permit held across cheap local work is capacity you paid for and left idle; and (c) that acquire has a timeout, so a caller waiting for a permit fails with a clear error instead of waiting forever behind a stuck holder.
Note also what is *not* here: no thread pool. A limit and a pool are different tools that people conflate. A pool bounds concurrency by bounding workers, which also decides where the work runs. A semaphore bounds concurrency without moving the work anywhere — it works on the caller's own threads, or over ten thousand async tasks on a single-threaded event loop where a "worker count" would be meaningless. The correct answer to "how do I limit my fan-out?" is almost always this, not a second pool.
- Guard the narrowest region that touches the constrained resource, not the whole task.
- Release in
finally, always. A leaked permit is a permanent, silent capacity reduction. - Give acquire a timeout so "the limiter is stuck" is an error, not a hang.
- A semaphore limits concurrency without deciding where work runs; a pool decides both. They are not substitutes.
1const limit = new Semaphore(50) // models the partner's 50-connection ceiling2 3async function enrich(record: Record): Promise<Enriched> {4 const local = prepare(record) // cheap and local — OUTSIDE the limit,5 // holding a permit for this wastes capacity6 await limit.acquire({ timeout: 5_000 }) // bounded wait: a stuck holder must not7 // silently stall every other caller8 try {9 return await partnerApi.enrich(local) // the ONLY thing the permit protects10 } finally {11 limit.release() // release on success, throw, and cancel.12 // Without finally, one thrown error leaks13 // a permit permanently and the ceiling14 // silently drops 50 -> 49 -> 48 -> ...15 }16}17 18// Fan out all 10,000. At most 50 are inside partnerApi.enrich() at any instant;19// the other 9,950 are cheap suspended tasks waiting on the semaphore, not threads.20const results = await Promise.allSettled(records.map(enrich))21// ^ allSettled, not all: with 10k tasks, 'all' rejects on the22// first failure while 9,999 keep running unobserved.23// See [[promise-all-and-gather]] and [[orphaned-tasks]].Where the limit goes, and who it protects
A concurrency limit placed in the wrong layer protects the wrong thing. The same fifty-permit semaphore means something entirely different depending on whether it wraps one caller's loop, a shared client object, a whole service instance, or the fleet.
The most common mistake is a per-call-site limit that everyone believes is a global one. Six code paths each politely bounded to 50 give the partner 300 concurrent connections, and every author can point at their own limit and say it works. If the limit exists to protect a *shared* resource, it must live at the same scope as the sharing — usually on the client object that owns the connection, not at the call site.
The second most common is a per-instance limit under an autoscaler. Fifty permits per instance times twelve instances is 600 concurrent calls, and the number changes whenever the scaler moves. If the downstream ceiling is global, either the limit belongs downstream (a server-side admission control, which is what perf's concurrency-limits lesson describes from the receiving end) or the per-instance number has to be derived from the fleet size and the ceiling together — and both of those are worse than asking the downstream to enforce its own limit and returning 429s you respect.
Which brings the real design rule: put the limit as close to the constrained resource as you can, and make everything above it respect the resource's own signal. A client-side limit is an optimisation that keeps you from generating load that will be rejected; the authoritative limit belongs to whoever owns the resource.
| Placement | What it bounds | Protects | Fails when |
|---|---|---|---|
| Inside one call site / loop | That one fan-out | Nothing shared — only this loop | Other call sites exist; the effective total is N × call sites |
| On the shared client object | All uses of that dependency in this process | The dependency, per process | The process is replicated; total is N × instances |
| Per service instance (a shared limiter) | Everything this instance sends anywhere | This instance's own resources (memory, sockets) | It is used as a proxy for a downstream limit it cannot see |
| At the resource itself (server-side admission control) | All callers, everywhere | The actual constrained resource, authoritatively | Callers do not respect rejections and retry immediately, amplifying load |
The leaked permit, and why the ceiling decays
The invariant has two halves and everyone remembers the first. "At most N inside" is easy. "Every permit released exactly once" is where the bug lives, and it produces the most distinctive failure signature in this module: a system that works, then works a bit less well, then eventually processes nothing, with no error at the moment of damage.
The schedule below traces it. A task acquires, throws before the release line, and the permit is simply gone. Nothing observes this. The effective ceiling drops from 50 to 49. The next occurrence takes it to 48. Because the failure rate is low, the decay is slow — which is precisely why it is diagnosed as "the partner got slower" rather than as a leak, and why it survives weeks of monitoring.
The same shape appears with a double release, running the other direction: a permit released twice raises the ceiling to 51, then 52, and the limit quietly stops limiting until the downstream starts returning 429s at a concurrency you believed was impossible. Both are variants of the same discipline failure, and both are prevented by never writing acquire and release as separate statements — use the language's scoped construct (RAII in C++, with in Python, a withPermit(fn) helper in JS/TS) so the release cannot be skipped or duplicated.
| # | Task 1 | Task 2 | Semaphore | State |
|---|---|---|---|---|
| 1 | · | · | initialised with 50 permits | available=50 held=0 in circulation=50 |
| 2 | acquire → granted | · | · | available=49 held=1 in circulation=50 |
| 3 | partner call throws ConnectionReset — release is after the throw, not in finally | · | · | available=49 held=0 in circulation=49 ✕ A permit vanished: nobody holds it and nobody can acquire it. The ceiling is now 49 and no signal was emitted. |
| 4 | · | acquire → granted | · | available=48 held=1 in circulation=49 |
| 5 | · | completes, release | · | available=49 held=0 in circulation=49 |
| 6 | retry, acquire, throws again | · | · | available=48 held=0 in circulation=48 ✕ Second permit lost. Retries on a failing dependency accelerate the decay exactly when the dependency is already unhealthy. |
| 7 | · | · | after ~6 hours and a few hundred transient errors | available=0 held=0 in circulation=0 ✕ No permits remain. Every acquire blocks forever; throughput is zero; every task is WAITING and no exception is ever thrown. |
| 8 | · | acquire → blocks indefinitely (no timeout configured) | · | queued waiters=9950 error rate=0% throughput=0/s |
with, a withPermit helper) makes the schedule unrepresentable; a finally makes it unlikely; separate statements make it inevitable.Key points
- A concurrency limit is a design decision made in advance, not a mitigation applied after an incident.
- Guard only the region touching the constrained resource; a permit held across cheap local work is idle capacity.
- Release in a scoped construct so the exception, cancellation and early-return paths cannot skip it.
- The limit must live at the same scope as the resource it protects: per-call-site limits multiply, per-instance limits multiply by the autoscaler.
- The authoritative limit belongs to the resource owner; a client-side limit only avoids generating load that would be rejected.
- A leaked permit is a permanent silent capacity loss whose signature is falling throughput with a zero error rate.
- Picking N: start from the downstream's stated or measured ceiling, not from your own core count.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A counting semaphore is initialised with N permits, where N models the capacity of the constrained resource.
- • Each task acquires a permit before entering the guarded region; if none is available the task waits (blocks a thread, or suspends if the runtime is async).
- • Exactly one task per permit executes the guarded region; the rest are parked in the semaphore's wait set.
- • On exit — normal, exceptional or cancelled — the permit is released and one waiter is woken.
- • The wait itself is bounded by a timeout so a stuck holder produces an error rather than an unbounded stall.
- • N is chosen from the downstream ceiling and validated by watching downstream saturation, then re-validated when that ceiling changes.
- • 50 tasks acquire, task 51 waits; task 7 completes and releases; task 51 is woken and proceeds — the intended schedule, and the reason the partner never sees 51 connections.
- • Task 1 acquires and throws before release; the permit is never returned; the effective ceiling is 49 forever, with no error and no metric change.
- • A task releases twice (release called in both a success branch and a finally); the ceiling rises to 51 and the limit silently stops holding.
- • A task is cancelled while parked in acquire; if the cancellation path does not distinguish "was granted" from "was waiting", it releases a permit it never held — the same double-release from a different direction.
- • Two tasks each need two permits and take them in different orders; each holds one and waits for the other — a deadlock built from a limiter, which is why multi-permit acquisition needs a single atomic acquire(2) or a fixed order (Lock Ordering).
- • All 50 permits are held by tasks blocked on a partner that has stopped responding; with no acquire timeout every one of the remaining 9,950 tasks waits forever and the process reports perfect health.
- • Guaranteed: at most N tasks are inside the guarded region simultaneously, as long as every acquire/release is balanced.
- • Guaranteed: the *count* is maintained atomically; concurrent acquires cannot both succeed on the last permit.
- • NOT guaranteed: fairness. A plain semaphore may hand the freed permit to a newly arriving task rather than the longest waiter, so a task can starve (Starvation, Fairness).
- • NOT guaranteed: any bound on how long a task waits. That is the acquire timeout's job, and it is opt-in.
- • NOT guaranteed: mutual exclusion over the data the tasks touch. A semaphore of 50 permits 50 concurrent writers; if they share state you still need Mutexes: What They Protect and What They Do Not.
- • NOT guaranteed: that N matches the resource. The permit count is your model of the downstream ceiling and can be stale, wrong, or multiplied by replication.
- • NOT guaranteed: protection from a burst of *arrivals*. A concurrency limit bounds simultaneity, not rate — 50 concurrent calls averaging 10 ms is 5,000 calls per second, which may violate a rate limit that a concurrency limit cannot see.
- • The permit counter is a single contended cache line; at very high acquire rates on many cores it becomes the bottleneck itself, and a sharded or per-core limiter is the answer.
- • Waiters queue on the semaphore's wait set; a release wakes one, but implementations that wake all produce a thundering herd on the counter (Thundering Herd).
- • The real contention is downstream and intentional: tasks 51 through 10,000 are waiting because you decided they should.
- • Holding a permit across an unrelated blocking call (a log flush, a DNS lookup, a lock) converts unrelated latency into limiter occupancy and shrinks effective capacity without shrinking N.
- • Permit leak on the error path: permanent capacity decay ending in a total stall with a zero error rate.
- • Double release: the ceiling silently rises and the limit stops limiting.
- • Deadlock from multi-permit acquisition in inconsistent order, or from a task holding a permit while waiting for work that needs a permit.
- • Starvation of a waiter under an unfair semaphore.
- • Effective-limit multiplication: N per call site, or N per instance under an autoscaler, giving a total nobody intended.
- • Rate-limit violation despite a correct concurrency limit — simultaneity and rate are different quantities.
- • Unbounded acquire waits turning a downstream stall into a fleet-wide hang with healthy-looking metrics.
- • Any fan-out over a collection whose size is data-dependent:
mapover 10,000 records is unbounded concurrency wearing a functional hat (Unbounded Concurrency). - • When a downstream has a hard, known ceiling — connections, quota, licences — and generating load above it produces rejections rather than throughput.
- • When memory per in-flight task is significant, so simultaneity, not CPU, is the resource being protected.
- • On an async runtime, where you want thousands of tasks in flight but only tens inside one specific dependency — a pool cannot express that and a semaphore can.
- • As the practical answer to a parallel fan-out that overloads a shared dependency (Parallelism Moves the Load Downstream, Fan-Out / Fan-In: One Request Becomes N).
- • When the real constraint is rate rather than simultaneity — a token bucket is the right tool and a semaphore will not stop you exceeding a per-second quota.
- • When tasks are heterogeneous: one permit for a 5 ms call and one for a 5-minute call treats unequal costs as equal, and the long calls squeeze out the short ones.
- • When the limit is set from local reasoning (cores, "feels safe") rather than the downstream's actual ceiling — an arbitrary number that constrains throughput and protects nothing.
- • When it hides an architecture problem: needing a limiter on ten different call paths to the same service usually means the calls should be batched (Batch APIs and Partial Failure in API design is the alternative worth reading).
- • Permits available over time. A monotonically declining floor is a leak; this single graph would prevent most permit-leak incidents.
- • Acquire wait time at p50/p99 — how much of your latency is your own limiter, which is the number that tells you whether N is too small.
- • Concurrent in-flight count as observed *downstream*, compared against N × instances. Divergence means the limit is not where you think it is.
- • Acquire timeout count: nonzero means holders are stuck, and it is the early warning before the stall.
- • Downstream rejection rate (429s, connection refusals) — if it is above zero with a limiter in place, either N is too high or the limit is multiplied.
- • Throughput against error rate together: falling throughput with a flat error rate is the leaked-permit signature and nothing else looks like it.
- • N becomes a tuned parameter coupled to a dependency's capacity, needing an owner, a rationale and a re-check when that dependency changes.
- • Every acquire site must be exception-safe and cancellation-safe, which is a discipline the type system will not enforce unless you build a scoped helper.
- • Waiting is now a state your system can be in, so the observability surface grows: wait time, timeouts, and available permits all need to be visible.
- • Multi-resource limiting introduces ordering requirements between limiters, with the same deadlock analysis as multiple locks.
- • A limiter interacts with retries: a retry after a timeout is a *new* acquire, so a retrying client under load acquires more aggressively than a patient one.
- • A dedicated pool of N workers, when you also want the work to run somewhere specific — the limit and the execution site are decided together.
- • A bounded queue with N consumers, when you want the excess work to be visible and droppable rather than parked in a wait set (Bounded vs Unbounded Queues).
- • A token-bucket rate limiter, when the constraint is requests per second rather than concurrent requests. Frequently you need both.
- • Batching: one call carrying 500 records instead of 500 concurrent calls removes the need for the limit entirely, and is usually the better engineering.
- • Chunked sequential processing (
forover slices of 50 withPromise.allper slice) — cruder, since a slice runs at the speed of its slowest member, but it needs no primitive and no release discipline. - • Respect the downstream's own signal: honour 429s with backoff and let the resource owner enforce its limit authoritatively.
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
10,000 tasks, N permits
One request, N downstream calls
What people believe, and what is true
We limit concurrency to 50, so we cannot exceed the partner's rate limit.
Concurrency and rate are different quantities. 50 concurrent calls averaging 10 ms is 5,000 requests per second. A concurrency limit cannot see rate.
Each service replica limits itself to 50, so we send at most 50.
You send at most 50 × replicas, and the autoscaler changes that number without telling you. Per-instance limits are not global limits.
A semaphore of 1 is just a mutex.
It has the same count and different semantics: a semaphore is not owned, so any thread may release one another acquired, and it is not reentrant. That difference is a feature for signalling and a hazard for mutual exclusion (Semaphore versus Mutex: Not the Same Primitive).
The limiter is a safety net; if it is too low we just get slower.
A limiter with a leak or an unbounded acquire wait converts downstream slowness into a total stall with no errors — a strictly worse failure than the one it prevented.
Go deeper
Overview
Hand out N tickets. A task needs a ticket to do the expensive thing and gives it back afterwards, so at most N are doing it at once.
Practical
Guard the narrow region, release in a scoped construct, bound the acquire wait, and set N from the downstream ceiling. Graph available permits.
Advanced
Place the limiter at the scope of the resource it protects. Under replication, prefer server-side admission control over per-instance arithmetic, and remember that concurrency limits do not bound rate.
Internals
The permit count is an atomic integer with a wait set; acquire is a decrement-if-positive CAS loop falling back to a park, and release is an increment plus a wake. At very high acquire rates that single cache line is itself the bottleneck.