The question this answers
Which of my single-machine concurrency intuitions survive when the state is on another machine?
Two application instances that must ensure a nightly report is generated exactly once, when both wake at 02:00 and both believe they should do it.
Nothing in memory — the two instances share no address space. The only shared state is whatever they both talk to: a database row, a lease record, a coordination service.
The report is generated exactly once per night, and no two instances believe simultaneously that they are the one generating it.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Three things you lose at the machine boundary
The first is shared memory. A mutex works because both threads can see and modify the same bytes, and the hardware provides an atomic operation over them — Atomics: What Is Actually Indivisible and Mutexes: What They Protect and What They Do Not. Across machines there are no shared bytes. Every "shared" value is a copy, and every update is a message to whoever owns the authoritative copy.
The second is bounded, reliable communication. A function call either happens or the program crashes. A network request has a third outcome: you sent it, and you do not know whether it arrived, whether it was processed, or whether the response was lost on the way back. That third outcome has no analogue in single-machine concurrency and it is the source of most distributed difficulty.
The third is failure atomicity of the participant. A thread that dies takes the process with it, so its locks are released with the address space. A node that dies while holding a lock leaves the lock held, by a holder that will never release it — and from the outside, a dead node and a slow node look identical. That last sentence is the crux: you cannot distinguish "crashed" from "paused for eight seconds", so any protocol must be correct under both.
| Concept | Single machine | Across machines | What breaks |
|---|---|---|---|
| Shared state | Bytes both threads can address | A copy on each side plus messages to an owner | Every read is potentially stale by the network round trip |
| Mutual exclusion | An atomic instruction on a memory word | A protocol: acquire request, grant, renew, release | The grant can be lost; the holder can die still holding it |
| "The lock is held" | A fact, readable instantly | A claim about the recent past, from a message that took time to arrive | The holder may have already lost it while you were reading |
| Release on crash | Automatic — the process dies with its locks | Nothing releases it; the lock is held by a corpse | Requires an expiry (a lease), which introduces a whole new failure mode |
| Detecting a stuck participant | Thread state, visible in a dump | Timeout only — and a timeout cannot distinguish slow from dead | You must choose: wait forever (safe, no progress) or assume dead (progress, possible double execution) |
| Ordering | A memory model gives rules | No global clock; messages arrive out of order | Causality must be carried explicitly, not inferred from timestamps |
The lease, and the failure it introduces
Because nothing releases a lock held by a dead node, distributed locks are almost always *leases*: the grant expires after a fixed duration unless renewed. That solves the corpse problem and creates a new one, because expiry is judged by the coordination service's clock while the holder is doing work measured by its own — and a holder that is paused (garbage collection, a hypervisor stall, a scheduler starvation, a network partition) can be past its expiry without knowing it.
The schedule below is the canonical result. Instance A holds a lease, pauses for longer than the lease duration, the lease expires, instance B acquires it legitimately and starts work, and then A resumes still believing it holds the lease. Two nodes now both believe they are the exclusive worker, and the coordination service did nothing wrong. This is not a bug in the lease service; it is a consequence of not being able to distinguish paused from dead.
The mitigation is a *fencing token*: the lease grant carries a monotonically increasing number, and the resource being protected rejects any operation carrying a token lower than the highest it has seen. That pushes enforcement to the resource, which is the only place it can actually be enforced — which is itself the important lesson, and the one that carries into A Mutex on Server A Does Nothing About Server B.
| # | Instance A | Lease service | Instance B | Report storage | State |
|---|---|---|---|---|---|
| 1 | acquire lease("nightly-report", ttl=30s) -> granted, token=17 | · | · | · | holder=A token=17 A believes=holds |
| 2 | begin generating the report | · | · | · | holder=A token=17 |
| 3 | process pauses (GC / hypervisor stall / partition) for 45s | · | · | · | holder=A token=17 A believes=holds |
| 4 | · | lease ttl elapses; lease released | · | · | holder=none token=17 |
| 5 | · | · | acquire lease -> granted, token=18 | · | holder=B token=18 A believes=holds |
| 6 | · | · | begin generating the report | · | holder=B token=18 |
| 7 | resumes; still believes it holds the lease | · | · | · | holder=B token=18 A believes=holds ✕ At most one instance is generating the report. Two are, and both are following the protocol correctly. |
| 8 | writes report.pdf (token 17) | · | · | · | report.pdf=from A token=18 |
| 9 | · | · | writes report.pdf (token 18) | · | report.pdf=from B token=18 |
| 10 | · | · | · | WITH FENCING: storage rejects any write with token < 18 | report.pdf=from B highest token seen=18 |
What to do instead of reaching for a distributed lock
The strong recommendation is to avoid needing exclusion at all, because every distributed locking scheme carries the failure above in some form. Three strategies do that, and they are ordered by how much they simplify.
First, make the operation idempotent and stop caring how many times it runs. If generating the report twice produces the same artefact at the same key, the whole problem disappears — and idempotency is a property you can test, unlike a lock protocol whose failure mode is a rare timing window. Second, make the resource itself enforce uniqueness: a unique constraint on (report_date) means the second writer is rejected by the database with no coordination anywhere. That is The Database Solves Concurrency For Its Data, Not For Your Memory's point applied here — the database is a coordination service you already run. Third, partition the work so no two nodes ever contend: assign report generation deterministically by hash, and there is no exclusion problem because there is only one candidate.
When exclusion genuinely is required, use a purpose-built coordination service rather than building one, use leases with fencing tokens, and — critically — write down what happens when the lease is lost mid-operation. That last question is the one that separates a design from an aspiration, and if the answer is "that cannot happen", the design is not finished.
Key points
- Across machines you lose shared memory, reliable bounded communication, and automatic lock release on crash. Each loss changes what a lock can mean.
- The third outcome of every remote call — sent, and you do not know what happened — has no single-machine analogue and causes most distributed difficulty.
- A dead node and a slow node are indistinguishable from outside, so every protocol must be correct under both readings.
- Distributed locks are leases because nothing releases a lock held by a corpse; leases in turn allow a paused holder to believe it still holds.
- Fencing tokens work because enforcement moves to the resource — exclusion is real where the write lands, not where the lock is granted.
- The best answers avoid exclusion entirely: idempotent operations, a uniqueness constraint at the resource, or deterministic partitioning.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A node requests a lease from a coordination service; the grant carries a duration and, in good designs, a monotonically increasing token.
- • The holder performs work and periodically renews. If renewal fails or is delayed past the duration, the service considers the lease free and may grant it to another node.
- • The former holder cannot be notified reliably — it may be partitioned or paused — so it may continue believing it holds the lease.
- • The protected resource records the highest token it has accepted and rejects operations carrying a lower one, which makes the exclusion enforceable rather than merely advisory.
- • On the failure side, callers must treat every remote outcome as three-valued: succeeded, failed, or unknown — and the unknown case is where retries and idempotency keys become mandatory.
- • A acquires a lease and pauses for longer than its TTL; B acquires legitimately; A resumes and writes. Both followed the protocol; without fencing, the last writer wins and one node's work is silently discarded.
- • A completes the work and its release message is lost. The lease expires anyway after its TTL, so correctness holds — but the resource was unnecessarily unavailable for the remainder of the duration.
- • A network partition splits the cluster: nodes on each side can reach a different subset of the coordination service. Whether both sides can be granted the lease depends entirely on the service's consistency guarantees, which is why "a distributed lock" built on a best-effort store is not one.
- • A request to increment a remote counter times out. The caller retries. The original request had in fact been applied, and the counter is now 2 — the classic at-least-once duplicate, and the reason idempotency is the load-bearing property here.
- • A lease guarantees a bound on how long a failed holder can block progress. It does not guarantee that only one node believes it holds the lease.
- • A fencing token guarantees that stale holders' writes are rejected *by resources that check it*. A resource that ignores the token provides no exclusion at all.
- • A coordination service can guarantee at most one grant at a time only if it is itself consistent under partition; a cache-like store cannot make that promise regardless of its API.
- • No timeout can distinguish a slow node from a dead one. Every design must choose between safety (wait, no progress) and liveness (assume dead, risk duplication), and that choice is not eliminable.
- • At-most-once delivery is not achievable over an unreliable network without additional state; at-least-once plus idempotency is the practical form of exactly-once.
- • Every acquisition is a network round trip, so the cost is microseconds-to-milliseconds rather than nanoseconds, and holding across remote calls is far more expensive than it looks.
- • A single coordination service becomes a shared dependency for every lock, and its availability bounds the availability of everything that locks.
- • Renewal traffic scales with the number of held leases, and a renewal storm after a partition heals can overwhelm the service.
- • Lease durations trade contention against recovery time: short leases recover fast and renew constantly; long leases are cheap and block progress for longer after a failure.
- • Split-brain: two nodes both believe they hold exclusive access, and both act. The signature failure of this whole area.
- • Lock held by a corpse, when a lock without expiry is used and its holder dies.
- • Duplicate execution from at-least-once retries after an unknown outcome, producing double charges, double emails and double writes.
- • Lost work: without fencing, the last writer wins and an earlier, correct result is silently overwritten.
- • Coordination-service unavailability turning into total system unavailability, because every path takes a lock.
- • Clock-based reasoning failing under skew: two nodes' clocks disagree and expiry decisions diverge — which is why monotonic tokens, not timestamps, are the enforcement mechanism.
- • Recognizing early that a requirement crosses a machine boundary, so the design starts with idempotency rather than reaching for a mutex that cannot work.
- • Choosing between the four strategies deliberately, since three of them avoid the failure modes above entirely.
- • Reviewing existing code for the assumption that a process-local lock or a process-local counter is global — see A Mutex on Server A Does Nothing About Server B.
- • Introducing a distributed lock when idempotency or a uniqueness constraint would have sufficed, buying a coordination dependency and a split-brain failure mode for nothing.
- • Taking a distributed lock on a hot path, where a network round trip per operation dominates the operation itself.
- • Building a lock on a store with no consistency guarantee under partition, which produces a system that works in testing and splits in production.
- • Lease acquisition latency and failure rate, which is now a network-dependent path in your critical section.
- • Renewal failure count, and how often a lease expires while its holder is still alive — the paused-holder signal.
- • Rejected fencing tokens, which directly counts how often a stale holder attempted a write. A non-zero count proves the failure mode is real in your system.
- • Duplicate-execution rate for operations guarded by exclusion, measured at the resource rather than inferred from the lock.
- • Coordination-service availability, since it bounds the availability of everything that depends on it.
- • A new operational dependency with its own quorum, failure modes and upgrade path, on the critical path of whatever it guards.
- • Every guarded operation needs an answer to "what if the lease is lost mid-operation", which is design work that cannot be skipped.
- • Idempotency keys, deduplication windows and retry policies become part of the contract of every operation — Idempotency Keys: The Mechanism.
- • Testing requires fault injection: partitions, pauses and clock skew, none of which occur in a normal test run.
- • Idempotent operations, so repetition is harmless and coordination is unnecessary — the single highest-leverage move available.
- • A uniqueness constraint or conditional write at the resource, which enforces exclusion exactly where it can actually be enforced — The Database Solves Concurrency For Its Data, Not For Your Memory.
- • Deterministic partitioning by key, so only one node is ever a candidate and there is nothing to contend for.
- • A queue with a single consumer per key, converting exclusion into ordering — Message Passing and The Actor Model.
- • Accepting duplicates and reconciling afterwards, which is often cheaper than preventing them and is a legitimate engineering choice.
What people believe, and what is true
A distributed lock is a mutex over the network.
A mutex is a fact about memory both parties can see. A distributed lock is a claim about the recent past, delivered by a message, held by a node that may already be gone.
A lease guarantees only one holder.
It guarantees only one *grant*. A paused holder past its TTL still believes it holds, and only a fencing check at the resource makes exclusion real.
We can detect the failed node and fail over safely.
No timeout distinguishes slow from dead. Every failover design is choosing between blocking forever and risking two active holders; there is no third option.