Concurrency by Design
Every piece of shared mutable state is a permanent tax on reasoning. Before reaching for a lock, ask whether ownership can be local, the data immutable, or the operation atomic — those remove the problem instead of managing it.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
Can this design avoid needing a lock at all, and what does it cost if it cannot?
Two requests updating the same order intermittently lose one of the updates. The proposed fix is a mutex around the update path. It is a service with four instances behind a load balancer.
Add a mutex around the read-modify-write. It is the textbook fix for a lost update and it makes the failing test pass.
It is the wrong scope. An in-process mutex serialises the four requests on one instance and does nothing about the four on the other three, so the bug becomes rarer and therefore much harder to diagnose (A Mutex on Server A Does Nothing About Server B in Concurrency).
- It is the wrong scope. An in-process mutex serialises the four requests on one instance and does nothing about the four on the other three, so the bug becomes rarer and therefore much harder to diagnose (A Mutex on Server A Does Nothing About Server B in Concurrency).
- It makes the shared state look protected, which is worse than obviously unprotected — the next reader trusts it.
- Locks compose badly. The second lock introduces the possibility of a deadlock, and nothing in the design says which order they must be taken in (Lock Ordering in Concurrency).
- It manages the symptom. The design still has state that several actors mutate, which is a cost paid on every future change to this code, not only on the update path.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The service runs as several processes, so an in-process lock protects nothing across instances (A Mutex on Server A Does Nothing About Server B in Concurrency).
- The database is the only thing all instances share, which makes it the only place a real invariant can be enforced (Enforcing Invariants).
- Latency matters: the fix cannot serialise all order updates.
- An order's total always equals the sum of its lines. Any interleaving that can produce a different result is a bug regardless of how rarely it happens (Invariants).
- No update is lost: a write that reports success is reflected in the stored state (The Lost Update, Step by Step in API Design).
- Whatever protects the invariant must work across processes, or it is a comment (The Thread-Safety Contract).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- Exactly one component owns each piece of mutable state and is the only thing permitted to change it (State Ownership).
- The store owns enforcing invariants that span instances, because it is the only shared thing (Consistency Boundaries).
- The interface owns saying whether concurrent use is allowed, so callers are not guessing (The Thread-Safety Contract).
- Whoever adds shared mutable state owns the reasoning burden it creates for everyone who reads the module afterwards (Local Reasoning).
- Draw the boundary so that mutable state does not cross it. State owned inside a module and mutated only through its interface needs no coordination from callers (Encapsulation).
- The consistency boundary is what must be updated atomically, and it should be as small as the invariant allows — usually one aggregate (Aggregates).
- Everything outside that boundary should be immutable or private, which is what makes local reasoning possible at all (Immutability).
The three questions, in order
The reflex is to reach for a synchronisation primitive, which manages the problem. The three questions try to remove it, and they are in order of how much they buy: local ownership removes sharing, immutability removes mutation, atomicity removes the multi-step window. Only if all three fail do you need a lock.
Look at what the shared object actually knows and does. The reasons-to-change list is the finding here, exactly as it is for any other responsibility problem.
- — The current state of every recently-touched order
- — Which orders are being modified right now
- — Nothing about the other three instances holding their own copy
- — Serves reads without hitting the database
- — Accepts mutations from any handler on this instance
- — Attempts to keep itself consistent with the store
- — Every handler that mutates an order
- — The database, eventually and unreliably
- — An assumption that this process is the only writer — which is false
- — The order model changes
- — A new handler needs to mutate an order
- — The eviction policy is tuned
- — The instance count changes — and nothing in the code says so
- — Someone adds a second field that must change atomically with the first
Five reasons to change and the fourth is not expressible in code at all: correctness depends on the deployment topology. Applying the three questions, the answer is not a lock — it is that this state should not be shared. Make the cache read-only and derived, and let every mutation be a conditional update against the store, which is the one thing all instances actually share.
Removing the window instead of guarding it
A lost update comes from a read-modify-write window. A lock keeps others out of the window; a conditional write removes the window by making the check and the write one operation.
The second version is not merely safer across instances — it is shorter, has no protocol for callers to follow, and cannot be broken by someone adding a new write path who did not know about the lock.
const locks = new Map<string, Mutex>()
async function addLine(orderId: string, line: Line) {
const m = lockFor(orderId) // per-instance only
await m.acquire()
try {
const order = await repo.byId(orderId)
order.lines.push(line)
order.total = sum(order.lines)
await repo.save(order) // last writer wins
} finally { m.release() }
}async function addLine(
orderId: string, line: Line, expected: Version,
): Promise<Added | Conflict> {
const order = addTo(await repo.byId(orderId), line) // pure
const rows = await repo.saveIfVersion(order, expected)
return rows === 1
? { kind: 'Added', version: expected + 1 }
: { kind: 'Conflict' } // the caller must decide
}
// UPDATE orders SET lines=?, total=?, version=version+1
// WHERE id=? AND version=?The left version is correct on one instance and silently wrong on two, and nothing in the code records that dependency on topology. The right version is correct on any number of instances because the check and the write are one operation in the only component all instances share. It is also more honest: Conflict is a real outcome that the caller has to handle, where the lock version quietly resolved it by letting the last writer win (Result Types).
What each option actually costs
None of these is free, and the common mistake is applying one uniformly. Immutability is superb for values and expensive for large collections; single ownership is superb for correctness and a throughput ceiling; optimistic concurrency is superb when conflicts are rare and pathological when they are not.
| Option | Simplicity | Performance | Testability | Operational | Note |
|---|---|---|---|---|---|
| Confine state to one request | No sharing, so no problem. Always the first choice, and available more often than it looks — most "shared" state is shared only because someone hoisted it to a module variable. | ||||
| Immutable values, replace rather than mutate | Removes data races and makes reasoning local. Does not give atomic multi-step updates, and copying costs on hot paths (Immutability). | ||||
| Optimistic concurrency in the store | The right default for a multi-instance service with rare conflicts. Callers must handle the conflict branch, and under contention the retry loop wastes work. | ||||
| Pessimistic row lock in the store | Correct across instances and simple to reason about. Serialises access, holds a transaction open, and gives you deadlocks to think about (Pessimistic Locking in Backend). | ||||
| In-process mutex | Correct on exactly one instance. Its operational score is the lowest here because its correctness depends on a deployment fact that no code expresses (A Mutex on Server A Does Nothing About Server B in Concurrency). |
caveat The scores compare mechanisms in a multi-instance service and cannot express the variable that decides between rows three and four: the conflict rate. At a fraction of a percent, optimistic concurrency is clearly better; on a hot row that a hundred writers contend for, the retry loop burns more than the lock would and pessimistic wins outright. Nor can they express the case where the invariant spans two aggregates — there, none of these rows is the answer and the model itself has to change (Consistency Boundaries).
How to build it
Most important first.
- Ask the three questions before any locking primitive. Can ownership be local? A value confined to one request needs no protection. Can the data be immutable? An immutable value can be shared by anyone with no coordination. Can the operation be atomic? A single conditional update in the store beats a read-modify-write in the application.
- Prefer a conditional write.
UPDATE orders SET total = ? WHERE id = ? AND version = ?with a check on the affected row count is a lost-update fix that works across every instance, needs no lock, and scales (Optimistic Concurrency: Versions and If-Match in Backend). - Confine state to a single owner and communicate by passing values rather than by sharing memory. That is the actor idea, and it works in a single process for the same reason it works across a network (The Actor Model in Concurrency).
- Where you genuinely need mutable shared state, put it behind an interface that makes the safe usage the only usage, rather than documenting a protocol callers must follow (Designing a Module Interface).
- If a lock is unavoidable, make its scope explicit, small and documented, and prefer the store's locks to your own (Lock Scope: What You Hold It Across in Concurrency).
- Design so there is less to interleave. Every request handler that touches only request-local state and one atomic store operation is a handler with no concurrency bugs available to it (Making an Existing Service Stateless in Backend).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- With shared mutable state: every future change to this module requires re-establishing which invariants hold under interleaving, and that reasoning cannot be done locally — it needs the whole set of concurrent actors. The cost of every subsequent change to the module goes up permanently, whether or not the change is about concurrency (Local Reasoning).
- With owned state and atomic operations: a new field costs one column and one line in the same conditional update. The reasoning stays local because there is nothing to interleave.
- Retrofitting a version column later costs a migration, a backfill, and a window where old and new writers coexist — one of the more awkward migrations there is, because correctness during the transition depends on every writer being upgraded (Expand and Contract).
- The permanent cost of optimistic concurrency: callers must handle a conflict result, which is one more branch at every write site.
- Immutability costs allocation and copying, which is genuinely significant on hot paths and in memory-constrained environments.
- Optimistic concurrency pushes conflict handling to callers, so every write site gets a branch, and under high contention the retry loop wastes work that a pessimistic lock would not.
- Single-owner designs can serialise throughput on one owner, which is a real scalability limit and the reason actor systems partition (Fan-in and Fan-out).
What can go wrong
- The lock is added at the wrong scope and the bug becomes rare rather than fixed, which converts a reproducible defect into a heisenbug (Heisenbugs: The Bug That Leaves When You Look at It in Concurrency).
- Optimistic concurrency is added and the conflict path is never handled: the update fails, the code ignores the row count, and the update is lost with a success response.
- The invariant spans two aggregates, so no single atomic operation can protect it, and the design needs to change rather than the locking (Consistency Boundaries).
- Everything is made immutable and the copying dominates, so a correctness fix becomes a latency incident (Allocation and Copies).
- A distributed lock is introduced and is assumed to be mutually exclusive, which it is not under partition without fencing (Distributed Locks: What They Are Actually For in Distributed Systems).
- Optimistic concurrency makes correctness depend on the store's atomicity guarantees, which is the right place for that dependency to live.
- Callers depend on the documented concurrency contract of every type they share, and an undocumented one is a dependency on an assumption (The Thread-Safety Contract).
- Immutable values remove a dependency entirely: there is nothing to coordinate, so no protocol to depend on (Immutability).
- "Add a mutex." Ask first whether the state needs to be shared. Most lost updates in a multi-instance service are not solvable with an in-process lock at all (A Mutex on Server A Does Nothing About Server B in Concurrency).
- "Immutability solves concurrency." It removes data races. It does not give you atomic multi-step operations, and the lost update here is about atomicity (The Atomicity Illusion in Concurrency).
- "Optimistic locking is a lock." It is a conditional write plus a conflict result. Nothing blocks, and callers must handle losing (Optimistic Concurrency: Versions and If-Match in Backend).
- "It only happens under load, so it is a performance problem." It is a correctness problem whose probability is a function of load. Rarity is not safety (Reasoning About Races: A Method, Not an Instinct in Concurrency).
- shared-state-coupling
- hidden-global-state
Testing it, and how it ages
- A concurrent test that runs two conflicting updates and asserts one wins and the other is told it lost. This is the test that actually exercises the mechanism (Where a Test Must Be Real).
- A property over random interleavings of valid commands, asserting the invariant after each (Property-Based Testing).
- A test that the conflict branch is handled: force a version mismatch and assert the caller sees a conflict rather than a silent success.
- Stress the path under real concurrency rather than reasoning about it. Interleaving bugs are not found by inspection (Stress Testing: A Test That Passed Once Proves Nothing in Concurrency).
- Systems drift towards more shared state as features are added, one convenience field at a time. The drift is invisible per commit and obvious per year (What Technical Debt Actually Is).
- Optimistic concurrency degrades under contention: it is excellent when conflicts are rare and pathological when they are common, at which point the design needs partitioning rather than a different lock (Optimistic vs Pessimistic in Concurrency).
- The eventual forcing function is scale: an in-process design that was fine on one instance stops being fine on twelve, and that transition is not gradual (What Changes at the Network Boundary).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat shared mutable state requires coordination, and that removing sharing or removing mutability removes the requirement, holds in every language and every runtime model.
- LANGUAGE-SPECIFICRust makes the ownership question a compile-time one: shared mutation is unrepresentable without an explicit synchronised type, so this whole lesson is enforced by the compiler. Go supplies channels and a culture of confining state to one goroutine but detects violations only at runtime under the race detector. Java and C# permit anything and rely entirely on convention. Same design advice, three wildly different costs of ignoring it.
- SCALE-SPECIFICOn a single process, an in-process lock is a genuine solution and everything here about atomic store operations is over-engineering. Across several instances it protects nothing, and the same code that was correct becomes silently wrong at the moment someone scales the deployment to two — which is a deployment change, not a code change, and that is what makes it dangerous.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — race detectors, stress harnesses and deterministic replay are how you find the interleavings this lesson tries to design away.
- — Programming Languages & Runtime Internals — memory models, happens-before edges and what a runtime guarantees about visibility decide whether a design is safe at all, and vary enormously between languages.