The question this answers
Which invariant does this particular mutex protect, and which of the guarantees I am assuming does it actually give me?
A rate-limiter bucket shared by every request handler: read the token count, refill by elapsed time, decrement if positive, and record the decision — with bucket.tokens and bucket.lastRefill both needing to agree.
bucket.tokens (a float) and bucket.lastRefill (a timestamp), in one object, reachable from every handler. The two fields must be updated together or the refill arithmetic double-counts elapsed time.
bucket.tokens never exceeds capacity and never goes below zero, and bucket.lastRefill always reflects the moment the current tokens value was computed — the two fields are consistent with each other at every observable instant.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
A mutex protects a region; you choose which invariant that region carries
The mechanism — a futex, a park/unpark pair, a spin-then-sleep adaptive lock — belongs to Operating Systems. Take it as given that acquire blocks until the holder releases and that at most one task is inside. The engineering content is entirely in the two decisions you make around it: *which region* and *which invariant*.
The rate-limiter below shows why the pairing matters. The invariant relates two fields, so the region must span both writes. It also spans the read of tokens, because the decision to decrement depends on it. What it must *not* span is the logging, the response construction, or anything that calls out of the process — all of which are commonly dragged inside by a synchronized keyword applied to a whole method.
Ownership is the other property worth naming explicitly, because it is what distinguishes a mutex from a semaphore (Semaphore versus Mutex: Not the Same Primitive). A mutex has an owner: the task that acquired it is the task that must release it. That is what makes reentrancy meaningful, what makes priority inheritance possible, and what makes "release from a different thread" an error rather than a technique.
1class TokenBucket {2 private tokens = CAPACITY3 private lastRefill = now()4 private readonly lock = new Mutex() // protects: tokens + lastRefill, together5 6 // WRONG: region too wide. The lock is held across an I/O call whose latency7 // is set by someone else, so this bucket's throughput ceiling is theirs.8 async allowBad(key: string): Promise<boolean> {9 return this.lock.withLock(async () => {10 this.refill()11 if (this.tokens < 1) { await auditSink.write({ key, allowed: false }); return false }12 this.tokens -= 113 await auditSink.write({ key, allowed: true }) // <-- 5 ms, inside the lock14 return true15 })16 }17 18 // RIGHT: the region is exactly the span across which the two fields disagree.19 async allow(key: string): Promise<boolean> {20 const allowed = this.lock.withLockSync(() => { // no await inside. deliberate.21 this.refill() // writes lastRefill AND tokens22 if (this.tokens < 1) return false23 this.tokens -= 1 // invariant restored here24 return true25 })26 await auditSink.write({ key, allowed }) // outside: different invariant27 return allowed28 }29 30 private refill() { // callers must hold the lock.31 const t = now() // Nothing in the type system32 const gained = (t - this.lastRefill) * RATE // enforces that; only this33 this.tokens = Math.min(CAPACITY, this.tokens + gained)34 this.lastRefill = t // comment does.35 }36}The schedule a mutex does not save you from
The most valuable thing to internalise about mutexes is the shape of the failure that survives them. A mutex makes each *region* indivisible. If your invariant spans two regions — because a refactor split them, because a helper acquires its own lock, or because the caller performs a sequence of individually-locked calls — the mutex is doing exactly what it promises and the invariant still dies.
The schedule below is the rate limiter with refill() and consume() each acquiring the lock separately, which is how this code usually ends up after someone makes refill public. Every access to both fields is under the lock. Mutual exclusion holds perfectly. And the bucket hands out more tokens than capacity, because the decision was made in one region and acted on in another.
That is the guarantee boundary, and it is worth stating as a sentence you can use in review: a mutex guarantees that no one is inside the region with you. It guarantees nothing about what happened between your two regions.
| # | Handler A — allow(key) | Handler B — allow(key) | State |
|---|---|---|---|
| 1 | lock; refill() → tokens = 1.0; unlock | · | tokens=1 holder=none |
| 2 | read tokens → 1.0, decide ALLOW [no lock held] | · | tokens=1 |
| 3 | · | lock; refill() → tokens = 1.0; unlock | tokens=1 holder=none |
| 4 | · | read tokens → 1.0, decide ALLOW [no lock held] | tokens=1 ✕ Two handlers have both decided to admit against a bucket holding one token. Every memory access so far has been under the mutex. |
| 5 | lock; tokens -= 1 → 0.0; unlock | · | tokens=0 holder=none |
| 6 | · | lock; tokens -= 1 → -1.0; unlock | tokens=-1 holder=none ✕ tokens is negative: the invariant 0 <= tokens is false, and two requests were admitted against a limit of one. |
What waiting costs, and where it shows up
A mutex converts a correctness problem into a queueing problem, and queueing problems have their own arithmetic. If the region takes H seconds to execute and requests arrive at rate R, the resource saturates at R = 1/H regardless of how many cores you have. At 80% of that ceiling, wait times begin to climb non-linearly; past it, the queue grows without bound and latency is limited only by your timeout. This is ordinary queueing theory applied to a lock — see What Contention Actually Costs and, for the diagnostic side, the performance domain's treatment of lock contention.
The timeline below shows four handlers against one bucket with a 2-tick region. The important reading is not that they wait — it is *where the time goes*: T4 spends six ticks blocked to do two ticks of work, so its latency is 4× its service time while the CPU is largely idle. Low CPU with high latency is the signature of lock contention, and it is why "add more cores" does nothing here.
The second thing the timeline shows is that a blocked task is not a spinning task. A well-implemented mutex parks the waiter so it consumes no CPU — which is good for the machine and terrible for your intuition, because the box looks healthy while the service does not. See Busy Waiting for the opposite trade-off and Lock Convoys for what happens when this queue never drains.
Key points
- A mutex guarantees at most one task inside the region. Everything else you might want from it, it does not provide.
- It has an owner: the acquiring task must release it. That is what separates it from a semaphore and what makes reentrancy and priority inheritance possible.
- A mutex protects an invariant only if the region spans the invariant. Two separately-locked regions with a decision in between are fully synchronized and fully broken.
- Throughput through a mutex is capped at 1/hold-time regardless of core count. More cores do not raise the ceiling; a shorter region does.
- Blocked tasks are parked, not spinning, so the machine looks idle while latency climbs. Low CPU plus high latency is the contention signature.
- It does not give fairness, does not give ordering between tasks, does not protect anything outside the region, and means nothing across process boundaries.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Acquire: if the lock is free, take ownership and proceed; if it is held, the task is added to a wait set and descheduled. Most implementations spin briefly first, on the bet that the region is short.
- • Inside the region, the holder may temporarily break the invariant — that is the entire point of excluding everyone else.
- • Release: ownership is dropped and one waiter is made runnable. Which one is unspecified in most implementations.
- • The woken waiter must be scheduled before it can proceed, so release-to-acquire latency includes a context switch, not just the handoff. See the OS treatment of context switching.
- • Reentrant variants track the owner and a recursion count, so the same task may acquire again; non-reentrant variants deadlock on that path instead.
- • A acquires, refills, checks, decrements, releases; B blocks and then observes 0 tokens and rejects. Correct: one region, one decision.
- • Split regions: A refills and releases; A decides ALLOW unlocked; B refills and releases; B decides ALLOW; both decrement. Tokens goes negative with every access locked.
- • A holds the lock across a 5 ms audit write; B, C and D block for 5, 10 and 15 ms. Correct, and the bucket now admits 200 requests per second instead of thousands.
- • A acquires lock L1 then L2 while B acquires L2 then L1: both block forever. Mutual exclusion working exactly as specified, in a cycle. See Deadlock and Lock Ordering.
- • A releases and three waiters wake; the one that wins is whichever the scheduler runs first, which may be the one that just released and re-acquired. Repeated indefinitely, that is starvation. See Starvation.
- • Guarantees: mutual exclusion within the region, and a happens-before edge from the release to the next acquire — so everything the previous holder wrote is visible to the next. That memory-visibility half is why a mutex also removes data races. See Happens-Before: The Edge That Makes a Write Visible.
- • Does NOT guarantee fairness. Most mutexes are unfair by design because barging is faster; a waiter can be skipped repeatedly. See Fairness.
- • Does NOT guarantee ordering between tasks. There is no FIFO promise unless the implementation explicitly says so, and fair locks cost throughput.
- • Does NOT guarantee anything about state outside the region — including state you read just before acquiring, which is stale the moment you acquire.
- • Does NOT guarantee that composing two locked operations is correct. This is the failure in the schedule above and the single most common mutex bug.
- • Does NOT guarantee anything across processes or machines. A mutex in one replica is invisible to the other five. See A Mutex on Server A Does Nothing About Server B.
- • Does NOT guarantee progress. A lock is precisely the primitive that lets a program stop making progress, which is what Deadlock is about.
- • Uncontended acquire is cheap — commonly tens of nanoseconds, a single atomic operation with no system call on modern implementations.
- • Contended acquire is expensive: a park, a context switch, and a wake. That is microseconds, three orders of magnitude more, which is why the first contended acquire changes the performance shape so abruptly.
- • Throughput ceiling is 1/hold-time. A 5 ms region caps the protected resource at ~200 operations per second no matter how many cores are available.
- • Queue growth past the ceiling is unbounded, so latency is limited only by timeouts. The characteristic incident is a slow dependency inside a region turning into service-wide latency. See Lock Convoys.
- • The lock cache line itself is contended: every acquire attempt writes it, invalidating it in every other core's cache. On a hot lock this is measurable on its own. See False Sharing: Different Variables, Same Cache Line.
- • Deadlock — two tasks each holding what the other needs. The failure a mutex makes possible and an atomic does not. See The Four Conditions.
- • Lock convoy — arrival rate exceeds 1/hold-time and the wait queue never drains, so latency grows without bound while CPU sits idle.
- • Starvation — an unfair lock repeatedly favours a barging acquirer and one waiter never runs.
- • Forgotten release on an error path — an early
returnor an exception between acquire and release leaves the lock held forever. This is why scope-bound forms (with,lock_guard,defer,withLock) exist and why manual acquire/release should be treated as a review flag. - • Composed-operation race — the schedule above. Every access locked, the invariant broken.
- • Priority inversion — a low-priority holder is preempted while a high-priority task waits on it. See Priority Inversion.
- • Deadlocking an event loop — taking a blocking mutex on a single-threaded runtime blocks the only thread, so the holder can never be scheduled to release it.
- • When the invariant spans several variables. This is the case an atomic cannot cover and the mutex's core competence.
- • When the region is short and contention is low — the uncontended path is cheap enough that clarity beats cleverness.
- • When you want the memory-visibility guarantee thrown in: a mutex is far harder to get subtly wrong than hand-rolled release/acquire atomics.
- • When the alternative is a hand-written lock-free algorithm and you have no measured reason to need one. See Atomics Are Not Magic.
- • When the region contains I/O. The hold time becomes someone else's p99 and your throughput ceiling with it. See Lock Scope: What You Hold It Across.
- • When the workload is read-dominated and reads are long — a mutex serialises readers who could all have proceeded. See Read/Write Locks, Honestly, with its own honest caveats.
- • When there is a single hot lock behind everything. Splitting the state so there are many locks is a modelling change with a far larger effect than any tuning.
- • When the operation is a single-variable read-modify-write — an atomic does it without a lock object, without a wait queue and without a deadlock surface.
- • On a single-threaded async runtime, where a blocking mutex is not slow but fatal, and the correct tool is an async-aware lock that yields.
- • Lock wait time at p99 and lock hold time at p99, reported separately. Long hold means the region is too big; short hold with long wait means it is acquired too often.
- • Contended-acquire count as a fraction of total acquires. Under a few percent, the lock is not your problem however much it looks like it.
- • The composite signal: CPU utilisation low, request latency high, thread dump shows many threads blocked on one monitor. That triple is lock contention and nothing else.
- • A thread dump or async task dump during the incident. It names the lock, the holder and the waiters directly. See Reading a Thread Dump.
- • Throughput against the predicted 1/hold-time ceiling. If measured throughput plateaus near that number, the lock is the bottleneck and shrinking the region is the only lever.
- • Every mutex adds an unwritten rule — which lock protects which fields — that the type system does not carry and that decays into a comment.
- • Two mutexes add an ordering obligation, and every additional lock multiplies the number of orderings that must be respected. This is where deadlock enters a codebase.
- • Acquire and release must be correct on every path including exceptions and early returns, which is why scope-bound forms are effectively mandatory.
- • A lock in a library becomes part of its contract: callers must know whether a callback they pass will be invoked with the lock held, which is one of the great sources of surprise deadlocks.
- • An atomic operation, when the invariant is one variable and one read-modify-write. No wait queue, no deadlock, no ordering obligation. See Atomics: What Is Actually Indivisible.
- • Do not share: partition the state so each task owns a piece, then combine. Removes the lock rather than tuning it. See Parallel Reduce.
- • Immutability plus a single reference swap: readers never block and there is nothing to hold. See Immutability as a Concurrency Strategy and Safe Publication: Handing Over a Finished Object.
- • Message passing to a single owner task, which serialises by construction with no lock object at all. See The Actor Model.
- • Optimistic control — read a version, compute, write conditionally, retry on conflict — when conflicts are rare enough that retries are cheaper than waits. See Optimistic vs Pessimistic.
A mutex buys correctness with throughput
Eight threads, one lock
How much of the task is inside the lock?
What people believe, and what is true
Adding a mutex makes this class thread-safe.
It makes each locked region mutually exclusive. Whether that yields correctness depends entirely on whether the region matches the invariant's span — see the negative-tokens schedule above.
Locks are slow.
Uncontended acquisition is a single atomic operation. What is slow is *contention* — the park, the context switch and the queue — which is a property of your region size and arrival rate, not of the primitive.
The waiting threads are wasting CPU.
A blocked thread is parked and consumes no CPU. That is exactly why the machine looks healthy during a contention incident and why utilisation is the wrong metric to watch.
The lock guarantees the waiting tasks are served in order.
Almost no mutex is FIFO by default; barging is faster and most implementations allow it. If you need order, you need an explicitly fair lock and you will pay throughput for it.
Go deeper
Overview
One task inside the region at a time. Everyone else waits. That is the whole promise.
Practical
Write down which fields each mutex protects and which invariant relates them. Keep the region to the invariant's span, keep I/O out, and always use a scope-bound acquire so no error path can skip the release.
Advanced
Throughput through a lock is 1/hold-time. Past that arrival rate the queue is unbounded. Contention is fixed by shrinking the region or by splitting the state into more locks — and the second option buys a lock-ordering obligation you must then document.
Internals
Modern mutexes are adaptive: a short spin (betting the region is shorter than a context switch) followed by a futex-style park that involves the kernel only on contention. The release also publishes memory: the release/acquire pair is what makes the previous holder's writes visible, which is why a mutex fixes both the logical race and the data race at once. See Happens-Before: The Edge That Makes a Write Visible and Memory Barriers Constrain Ordering, Not Caches.