The question this answers
How do I express "at most N of these at a time", and what happens to the permit when the task in the middle throws?
A report service running database queries against a pool of 20 connections, called by up to 1,000 concurrent request handlers.
The permit count itself, and the pool of 20 physical connections it stands for. The permit count is a proxy for a real, finite, external resource — that correspondence is the entire design.
At most 20 queries are in flight at any instant — permitsHeld + permitsAvailable === 20 at every instant, and every permit acquired is eventually released exactly once.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
A counter that stands for something real
A counting semaphore holds an integer. acquire() decrements it, blocking while it is zero; release() increments it and wakes a waiter. Operating Systems covers the implementation. What matters here is the modelling discipline: the count should be the size of a real constraint, not a number someone tuned.
That constraint is usually external and usually already known. The database accepts 20 connections. The image encoder needs 200 MB per job and the container has 4 GB. The partner API allows 50 concurrent requests. In each case the number exists before you write any code, and the semaphore is how you make your process respect it. When you cannot name what the number *is*, you are not resource-limiting — you are guessing, and the guess belongs in Bounding Concurrency where sizing is discussed honestly.
The connection pool is the canonical example because it is where the failure is most visible. Without a permit, 1,000 handlers all attempt to open a connection, the database refuses at its limit, and every handler receives a connection error. With a permit, 20 proceed and 980 wait — and the crucial design question becomes what "wait" means. An unbounded wait converts a capacity problem into a latency problem and then into a timeout storm; a bounded wait (tryAcquire with a timeout) converts it into a fast, honest rejection that a caller can retry or degrade around. See Backpressure.
1import asyncio, contextlib2 3# 20 because the database is configured for 20 connections.4# The number is not tuned; it is copied from the thing it represents.5DB_PERMITS = asyncio.Semaphore(20)6 7# WRONG: the release is on the happy path only.8async def run_report_bad(q):9 await DB_PERMITS.acquire()10 rows = await db.fetch(q) # if this raises, we never release.11 await DB_PERMITS.release() # 20 exceptions => pool permanently at zero.12 return rows13 14# RIGHT: scope-bound. The permit is returned on every exit path, including15# an exception and an asyncio.CancelledError from a client disconnect.16async def run_report(q):17 async with DB_PERMITS: # acquire ... release in __aexit__18 return await db.fetch(q)19 20# BETTER for a user-facing path: bound the wait, so a capacity problem21# surfaces as a fast rejection rather than as a 30-second timeout.22async def run_report_bounded(q):23 try:24 await asyncio.wait_for(DB_PERMITS.acquire(), timeout=0.250)25 except asyncio.TimeoutError:26 raise ServiceBusy('report queue full') # 503 + Retry-After, in 250 ms27 try:28 return await db.fetch(q)29 finally:30 DB_PERMITS.release()31 32# What the permit does NOT bound:33# - memory: 20 concurrent queries each streaming 500 MB is still 10 GB.34# - the WAITING queue: 980 handlers parked on acquire() still hold their35# request objects, sockets and buffers. Bound that queue separately.36# - anything in another process. Six replicas x 20 permits = 120 connections37# against a database configured for 20. See [[local-lock-not-distributed]].Twenty permits, a thousand arrivals
The timeline shows what a semaphore actually does to a burst. Five permits, eight arrivals, each query taking three ticks. Tasks 1 through 5 run immediately; 6, 7 and 8 wait and then run. Total throughput is capped at permits/duration — five queries per three ticks — and the waiting tasks contribute latency without contributing load. That is the trade the semaphore makes explicit: you convert a resource-overload failure into a queueing delay, on purpose.
The number to watch is not the permit count but the *wait time and the queue depth*. A semaphore that never has a waiter is not doing anything. A semaphore with a permanently non-empty queue is telling you the resource is undersized for the offered load, which is a capacity decision rather than a concurrency one — see the performance domain's treatment of pool saturation.
And note the second lane in the timeline: the resource itself is fully utilised the entire time. That is what "correctly sized" looks like. If the resource shows idle time while tasks are waiting, the permit count is lower than the real limit and you are throttling yourself.
The leaked permit
The failure that actually takes services down is not contention on a semaphore. It is a permit that was acquired and never released, because the task in the middle threw, was cancelled, or returned early on a path nobody tested. Each leak permanently reduces the pool by one. Twenty leaks and the pool is at zero forever — not slow, not degraded: zero, for the lifetime of the process, with the resource it protects completely idle.
The schedule below shows two leaks against a pool of three. What makes this failure so nasty operationally is its signature. The database shows near-zero connections and near-zero load. The application shows every request timing out. Every instinct says "the database is fine, so the problem is elsewhere", and the actual problem is a counter in your process that will never go back up.
The fix is structural, not vigilant: never write a bare `acquire()`. Use the scope-bound form the language provides — with/async with, try/finally, RAII, defer, using — so that every exit path returns the permit. A code-review rule that flags any acquire without a matching scope guard costs nothing and eliminates the entire class. The diagnostic is equally cheap: expose available permits as a gauge and alert when it stays low while the protected resource is idle. That combination — permits exhausted, resource idle — has exactly one cause.
| # | Report A — query succeeds | Report B — query raises | Report C — client disconnects, task cancelled | State |
|---|---|---|---|---|
| 1 | acquire → permits 3 → 2 | · | · | available=2 held=1 |
| 2 | · | acquire → permits 2 → 1 | · | available=1 held=2 |
| 3 | · | · | acquire → permits 1 → 0 | available=0 held=3 |
| 4 | query returns; release → permits 0 → 1 | · | · | available=1 held=2 |
| 5 | · | db.fetch raises QueryError — propagates past the release line | · | available=1 held=2 ✕ B holds a permit and has no code path left that will return it. The invariant "every acquire is matched by a release" is now permanently false. |
| 6 | · | · | client disconnects; task cancelled at the await — release never runs | available=1 held=2 ✕ A second permit is gone. Available capacity is now 1 of 3, and the count can never rise above 1 again. |
| 7 | next request: acquire → permits 1 → 0 | · | · | available=0 held=3 |
| 8 | · | next request: acquire blocks — permits 0, and 2 of 3 are leaked | · | available=0 held=3 ✕ Every subsequent request now waits for the single non-leaked permit. Throughput has fallen to one third and will fall to zero on the next leak. |
| 9 | · | · | DIAGNOSTIC: database reports 1 active connection, 5% CPU | available=0 held=3 |
Key points
- A semaphore is a counter with a waiting room:
acquiredecrements and blocks at zero,releaseincrements and wakes a waiter. - Its natural use is resource limiting, not mutual exclusion. The count should be the size of something real — connections, memory budget, a partner's concurrency cap.
- It converts a resource-overload failure into a queueing delay, deliberately. That is the trade, and it is usually a good one.
- Bound the wait. An unbounded
acquireturns a capacity problem into a timeout storm;tryAcquirewith a timeout turns it into a fast, honest 503. - The permit bounds only what it counts. It does not bound memory, does not bound the waiting queue, and means nothing across processes — six replicas of 20 permits is 120 connections.
- The failure that matters is the leaked permit: acquired, never released, because of an exception, a cancellation or an early return.
- Never write a bare
acquire(). Scope-bound acquisition on every path eliminates the entire class of leak. - The diagnostic signature of a leak is permits exhausted while the protected resource sits idle.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The semaphore holds a count and a wait queue.
acquireatomically decrements if the count is positive; otherwise the task is parked on the queue. - •
releaseatomically increments and, if the queue is non-empty, makes one waiter runnable. Which waiter is implementation-defined unless the semaphore is documented as fair. - • The count has no owner: any task may release, including one that never acquired. That is a feature for signalling and a hazard for resource limiting.
- • A permit is a promise about a resource elsewhere; nothing in the semaphore verifies that the promise is kept, which is why leaks are silent.
- • Bounded acquisition (
tryAcquire(timeout)) returns failure instead of parking, which is what makes rejection possible and backpressure expressible.
- • Twenty permits, 1,000 arrivals: 20 proceed, 980 park, and each release admits exactly one waiter. The pool is never oversubscribed and the database never sees a connection error.
- • No semaphore, 1,000 arrivals: all 1,000 attempt to connect, the database refuses beyond its limit, and every handler gets an error including the ones that would have succeeded.
- • Exception between acquire and release: the permit is never returned. Repeat once per error and the pool decays monotonically to zero.
- • Cancellation at an
awaitinside the region: same leak, and harder to spot because no exception is logged in the usual place. - • Unbounded waits under sustained overload: 980 tasks park, each still holding its request buffers and socket, and the process runs out of memory before the queue drains. The permit bounded the connections and not the waiters.
- • Six replicas each with 20 permits against a 20-connection database: each process respects its own limit and the database sees 120 attempts. The permit is process-local.
- • Guarantees that at most N tasks are between
acquireandreleaseat any instant, within this process. - • Does NOT guarantee mutual exclusion unless N is 1 — and even then it lacks a mutex's ownership, so a different task can release it. See Semaphore versus Mutex: Not the Same Primitive.
- • Does NOT guarantee fairness or FIFO ordering among waiters unless the implementation explicitly says so; a waiter can be repeatedly overtaken. See Fairness.
- • Does NOT guarantee that the permit is ever returned. That obligation is entirely on your code, and nothing detects a violation.
- • Does NOT bound the number of *waiters*, only the number of holders. The queue is unbounded unless you bound it separately.
- • Does NOT bound anything the permit does not count: memory, CPU, downstream fan-out, or the same resource accessed from another process.
- • The permit count is a single atomic cell, so a very hot semaphore contends on one cache line like any other atomic — usually irrelevant next to the resource it protects.
- • The real contention is the queue: waiters accumulate at arrival rate minus service rate, and the wait time follows straight from queueing theory. See Queueing: Why Systems Get Slow Before They Get Broken in performance.
- • A permit count set below the resource's true limit throttles you artificially, showing as waiters queueing while the resource reports idle capacity.
- • A permit count set above the true limit does not remove the constraint; it moves the failure from your queue to the resource's error path, which is strictly worse because the resource rejects rather than queues.
- • Permit leak — acquired and never released on an error, cancellation or early-return path. Monotonic, permanent, and cured only by a restart.
- • Pool exhaustion under load, with waiters piling up — a capacity problem correctly surfaced by the semaphore rather than caused by it.
- • Timeout storm from unbounded waits: every waiter times out at once, retries, and the herd re-arrives. See Thundering Herd.
- • Unbounded waiter queue exhausting memory while the guarded resource is comfortably within its limit.
- • Double release — releasing a permit that was never acquired, silently raising the effective limit above the resource's real capacity. The mirror of a leak and much harder to notice.
- • Deadlock by nesting: a task holding a permit waits for a second permit from the same semaphore, and with N tasks each holding one and needing two, nobody proceeds.
- • Cross-process overshoot — N permits per replica times R replicas against a resource sized for N.
- • Whenever a downstream resource has a real, known concurrency limit: a connection pool, a partner API, a licence count, a GPU, a memory budget expressed as concurrent jobs.
- • When you want overload to appear as a bounded queue plus fast rejection rather than as errors from a resource you do not control.
- • When fanning out to many downstream calls and needing to cap the multiplier. See Fan-Out / Fan-In: One Request Becomes N and Bounding Concurrency.
- • As a signalling device between producer and consumer, where the counting behaviour is the point — one permit per item produced. See Producer / Consumer.
- • As a substitute for a mutex. A binary semaphore has no owner, no reentrancy and no priority inheritance, and it can be released by a task that never acquired it.
- • When the limit is not a real constraint but a guess — a semaphore around a number nobody can justify is a throughput cap with no rationale, and it will be tuned by superstition.
- • When the wait is unbounded on a user-facing path, where it converts capacity pressure into latency and then into a retry storm.
- • When the real constraint is memory rather than concurrency: 20 concurrent 500 MB queries is 10 GB, and a concurrency permit says nothing about that.
- • When the resource is shared across processes, where a process-local count multiplies by the replica count.
- • Available permits as a gauge, sampled continuously. This one metric detects leaks, undersizing and oversizing.
- • Wait time at p99 and queue depth. Persistently non-empty means the resource is undersized for the offered load; always empty means the semaphore is not doing anything.
- • Acquire/release counts as separate counters. A monotonically growing difference is a leak, and this is the cheapest possible detector.
- • The correlation that names a leak unambiguously: permits at zero while the protected resource reports low utilisation.
- • Rejection rate when using a bounded acquire — this is the honest capacity signal and belongs on the service dashboard next to error rate.
- • Every acquire creates a release obligation on every exit path, including ones the compiler will not point out. Scope-bound forms discharge this and should be mandatory.
- • The permit count becomes a configuration value that must track the real resource; when the database is resized and the permit count is not, the two drift silently.
- • Bounded acquisition adds a rejection path, which adds a caller-side decision about degradation, retry and user-visible errors.
- • A process-local permit in a replicated service requires either a per-replica share of the true limit or a shared limiter, and both are decisions someone must make and write down.
- • A bounded work queue with a fixed pool of consumers, which limits concurrency and makes the waiting queue explicit and boundable in one structure. See Bounded vs Unbounded Queues and Worker Pools Beyond Threads.
- • The resource's own pool, when it has one — most database drivers already implement a connection pool with waits and timeouts, and adding a semaphore on top gives you two limits to keep in sync.
- • A rate limiter, when the constraint is requests per second rather than concurrent requests. The two are different constraints and a semaphore only expresses the second.
- • A token bucket or leaky bucket at the edge, when the goal is protecting a downstream partner rather than a local resource.
- • No limit at all, when the resource genuinely has none and the concurrency is naturally bounded by the caller — an unnecessary semaphore is a throughput ceiling you installed yourself.
10,000 tasks, N permits
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
The producer is faster than the consumer
What people believe, and what is true
A semaphore with one permit is a mutex.
It provides mutual exclusion and nothing else a mutex provides: no ownership, so any task can release it; no reentrancy; no priority inheritance; and no error when released by a task that never acquired. See Semaphore versus Mutex: Not the Same Primitive.
The semaphore limits how much load the service takes.
It limits holders, not arrivals. Nine hundred and eighty tasks parked on acquire still occupy memory, sockets and request state. Bound the queue separately or bound the wait.
Twenty permits means twenty connections to the database.
It means twenty per process. Six replicas is 120 against a database sized for 20 — the permit is local and the resource is shared.
We would notice a permit leak.
Its signature is the service failing while the protected resource looks perfectly healthy, and it disappears on redeploy. It is routinely misdiagnosed for weeks.