The question this answers
Can I restructure this so nodes act independently instead of agreeing first?
None universally. Coordination avoidance guarantees only what the restructured design guarantees, which is usually weaker: convergence rather than linearizability, eventual repair rather than prevention, or per-key rather than global serialization. The value is in making that weakening explicit and bounded, instead of assuming the strong guarantee was needed.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The whole technique turns on making the information a node needs locally available. Under partitioned ownership a node knows the full state of the keys it owns, which is all it needs. Under commutative operations a node needs to know nothing about others, because any order produces the same result. Under repair-later a node knows it may be wrong and that a detector will catch it. Avoidance is really the art of shrinking what a node must know down to what it can actually see.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Question one: can a violation be repaired?
Coordination prevents a bad state from ever existing. Repair permits it briefly and fixes it afterwards. Prevention is not automatically better — it is better only when the violation is unrepairable or when the repair is more expensive than the coordination.
Airlines overbook. Hotels overbook. Warehouses oversell and issue refunds. These are not sloppy systems; they are systems where the business has priced the repair and found it cheaper than the availability cost of preventing it. The engineering mistake is assuming the technical invariant is a business invariant without asking.
The test is concrete: what does the business currently do when this goes wrong? If the answer is "we refund and apologise", a repair path already exists, is already staffed, and probably already handles other causes of the same outcome. If the answer is "this has never happened and would be a catastrophe", you are looking at a real invariant.
| Violation | Repair | Verdict |
|---|---|---|
| Two users get the last seattypical | Refund, upgrade, rebook | Repairable — avoid coordination |
| Inventory oversold by 3typical | Backorder or cancel | Repairable — avoid coordination |
| Duplicate notification senttypical | None needed; mildly annoying | Repairable — avoid coordination |
| Two users get the same usernameassumption | Rename one — breaks their links and identity | Poor repair — coordinate |
| Account balance goes negativeassumption | Depends entirely on the product | Sometimes repairable (overdraft fee), sometimes not |
| Two nodes write the same file regionprotocol | None — the data is gone | Unrepairable — coordinate and fence |
Question two: do the operations commute?
If applying operations in different orders yields the same final state, there is no order to agree on, and coordination has nothing left to buy. add(x) to a set commutes with add(y). Incrementing a counter commutes with incrementing it again. Recording an event in an append-only log commutes with recording another, as long as you only ever read the whole set.
What does not commute is anything conditional on a global state: "decrement if the result is non-negative" does not commute with itself, because two nodes each seeing a balance of 10 and each decrementing by 8 both pass their local check and produce -6. The conditional is where the ordering requirement lives, and removing the conditional is often the actual design work.
A frequent and useful transformation: replace a check-then-act with a record-then-evaluate. Instead of "check inventory, then decrement", record the reservation as an event and let a single evaluator decide which reservations are honoured. The recording commutes; the evaluation happens once, in one place, off the critical path. CRDTs: Deterministic Merge, Not Correct Merge is the formal version of this idea for data structures.
1# coordinated: every request pays for agreement2acquire_lock("inventory:sku-42") # availability now coupled3 n = read("inventory:sku-42")4 if n <= 0: reject()5 write("inventory:sku-42", n - 1)6release_lock()7 8# commutative: requests never coordinate9append(reservations, {sku: "sku-42", user: u, at: t, id: uuid()}) # commutes10 11# one evaluator, off the request path, decides the truth12for r in reservations.new():13 if honoured_count("sku-42") < stock("sku-42"): confirm(r)14 else: waitlist(r) # the repair pathQuestion three: can ownership be partitioned?
This is the highest-leverage move in the module and the one that dissolves the most cases. If every key has exactly one owner node, that node makes decisions about the key alone, with no agreement, at local speed, with no availability coupling to any peer. A balance invariant per account needs no global coordination if all operations on that account route to one node.
The coordination has not vanished — it has moved to the *assignment* of ownership, which changes rarely and is exactly the kind of low-frequency, high-leverage fact Do You Actually Need Consensus? says consensus is for. You pay for agreement once per ownership change instead of once per request.
What this cannot do is protect invariants that genuinely span partitions. "Total inventory across all warehouses must not go negative" does not decompose by warehouse unless you also decompose the *stock*: give each warehouse a fixed allocation and let it decide locally within its allocation. That trick — partitioning the resource, not just the data — is what makes escrow and reservation schemes work, and it converts a global invariant into several local ones at the cost of some efficiency, since one partition can run out while another has spare.
- Node 1 — owns accounts a–h; decides alone
- Node 2 — owns accounts i–p; decides alone
- Node 3 — owns accounts q–z; unreachable
- Ownership map — consensus — changes rarely
- n1believes “I may decide about account "carol" without asking anyone”✓ and it is true
- n2believes “Node 3 being unreachable does not affect my accounts”✓ and it is true
- n3believes “I still own q–z”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
The honest residue
It would be dishonest to present avoidance as always available. Some invariants genuinely require global serialization, and no restructuring removes it. If a value must be unique across a namespace that cannot be partitioned, if a global total must never be exceeded and cannot be split into allocations, if an operation must observe the effects of every prior operation everywhere — you need coordination, and the work is to make it as narrow and as amortised as possible.
The CALM theorem gives the precise statement of the boundary: a computation can be executed without coordination if and only if it is monotonic — if adding more information never retracts a previous conclusion. "Has this set ever contained x?" is monotonic. "Does this set currently *not* contain x?" is not, because more information can change the answer from yes to no. Almost every invariant that resists avoidance resists it for exactly this reason: it contains a negation over a global state.
That is a genuinely useful test at design time. When an invariant refuses to decompose, look for the negation — "no other user has this name", "no other node holds this lock", "the total does not exceed" — and you will have found the thing that forces coordination.
Key points
- Ask three questions before coordinating: is a violation repairable, do the operations commute, can ownership be partitioned?
- Prevention is better than repair only when the violation is unrepairable or the repair costs more than the coupling.
- Conditionals on global state are what break commutativity; converting check-then-act into record-then-evaluate often removes them.
- Partitioning ownership dissolves the most cases: coordinate on the assignment, not on the decisions.
- Global resources can sometimes be partitioned into local allocations, converting one global invariant into several local ones.
- Some invariants really do require global serialization; CALM identifies them by their non-monotonicity — look for the negation.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • State the invariant precisely, including its scope: per key, per tenant, or global.
- • Ask whether a violation is detectable and repairable, and what the business already does about it today.
- • Ask whether the operations commute; if a conditional blocks commutativity, try to move the condition to a single evaluator.
- • Ask whether the state can be partitioned so one owner decides, or whether a global resource can be split into per-owner allocations.
- • If all three fail, look for the negation over global state — that is the part that genuinely needs coordination — and coordinate only that.
- • Whatever remains coordinated should be amortised into leases or tokens rather than paid per request.
- • The repair path is designed but never exercised, so it does not work when first needed.
- • Operations that were believed to commute do not, because of a conditional nobody noticed.
- • The partition key is wrong, so a significant fraction of operations span partitions and quietly re-acquire the coordination you removed.
- • Allocations are unbalanced: one partition exhausts its share while others hold spare, producing failures with global capacity remaining.
- • The ownership map itself becomes a hot dependency, reintroducing coupling at a different layer.
- • Repair path that never ran: an oversell reconciliation job has a bug and has been silently failing for weeks. The operator sees a growing backlog of unreconciled records and customer complaints long before any alert fires — the failure of a repair path is invisible unless you monitor the *delta*, not the job.
- • Cross-partition creep: an operation that was single-owner starts touching two partitions after a feature change. The operator sees latency and error rates on that endpoint diverge from the rest, and distributed transactions appearing in traces where none existed.
- • False commutativity: two "independent" increments turn out to share a conditional check. The operator sees a counter that is occasionally wrong by small amounts with no error anywhere — the hardest class of bug in this module to detect.
- • Allocation starvation: per-node inventory allocations run out unevenly and requests fail while global stock remains. The operator sees rejections concentrated on some nodes and idle stock on others.
- • Ownership churn: the ownership map changes frequently, so the "rare" coordination becomes routine and its cost returns. The operator sees ownership-change events correlating with latency spikes.
- • The goal is to reduce coordination to ownership assignment, which is rare, rather than to per-operation agreement, which is not.
- • Repair-later moves coordination off the critical path into a background process where its availability cost is invisible to users.
- • Commutative designs eliminate coordination for those operations entirely — the only genuinely free case in the module.
- • Partitioned ownership degrades per partition: a partition whose owner is unreachable is unavailable, and every other partition is unaffected. This is a far better failure profile than a global coordination point.
- • Commutative operations remain fully available during any partition and converge afterwards.
- • Repair-later designs stay available and accumulate a repair backlog whose size is the honest measure of the debt incurred.
- • Detect: monitor the *outcome* of the repair path — the count of unrepaired violations — not merely whether the job ran.
- • Contain: cap the exposure. A repair-later design should bound how far it can drift (a maximum oversell, a maximum backlog) and start refusing beyond it.
- • Recover: for a partition whose owner is unavailable, reassign ownership through the consensus-backed map, with fencing so the old owner cannot act.
- • Reconcile: run the repair, and record every violation so the rate is visible and can be argued about with business owners.
- • Verify: test the repair path in production regularly. An untested repair path is not a repair path.
- • Violation rate and repair rate as separate metrics; the gap is the real exposure.
- • Fraction of operations that span partitions — the metric that tells you whether your partitioning still fits the workload.
- • Ownership-change frequency, which should be low by design.
- • Allocation utilisation skew across owners, for escrow-style schemes.
- • Age of the oldest unrepaired violation.
- • High-frequency operations where the invariant is soft or repairable.
- • Workloads that partition naturally by tenant, user, account or shard key.
- • Multi-region designs where coordination would otherwise cost an inter-region round trip on every request.
- • Anywhere availability matters more than immediate precision, and precision can be restored.
- • Invariants with no acceptable repair — money that leaves the system, identity, safety-critical state.
- • When the repair path is more complex than the coordination it replaced, which happens more often than teams expect.
- • When "eventually consistent" is chosen as a default rather than derived from a stated invariant, leaving nobody able to say what the system guarantees.
- • Coordinate. When the invariant is hard and global, pay for it deliberately and narrowly — see Start From the Invariant, Not From the Architecture and Do You Actually Need Consensus?.
- • A single database with a constraint or a transaction: a coordination point most teams already run well, and the correct answer far more often than a distributed design.
- • Optimistic execution with compensation, where the operation proceeds and a failure triggers an explicit reversal. See Sagas: Trading Isolation for Availability and A Refund Is Not a Rollback.
- • Bounded staleness: coordinate periodically rather than per operation, accepting a known window of possible violation.
Restructuring instead of paying for agreement
| Fully coordinated | Per-warehouse allocation | Record then evaluate | |
|---|---|---|---|
| Guaranteeprotocol | The global total never goes negative, checked at one serialization point. | No warehouse exceeds its own allocation. The global total is respected by construction. | Nothing up front. A violation is possible and is repaired by a background evaluator. |
| Coordination per requestprotocol | One round trip to the authority. | None. The owning node decides alone. | None. |
| Under partitionprotocol | Unavailable on the minority side. | Each warehouse keeps selling its own stock; only its own partition is affected. | Fully available; the repair backlog grows. |
| How it failstypical | Total outage from partial failure; throughput capped by one node. | Allocation starvation — one warehouse rejects while global stock remains idle elsewhere. | Oversell that must be refunded, and a repair path nobody is watching. |
What people believe, and what is true
Avoiding coordination means giving up correctness.
It means changing which property you guarantee. A partitioned-ownership design can be perfectly linearizable per key, with no global coordination at all.
If it needs a lock, it needs a distributed lock.
It needs whatever makes the decision single-threaded. Routing all operations for a key to one node achieves that with no lock service and no availability coupling.
Eventual consistency is coordination avoidance.
Eventual consistency is one possible *result*. Avoidance is a design method, and partitioned ownership can avoid coordination while remaining strongly consistent per key.
Any invariant can be restructured if you are clever enough.
Non-monotonic invariants cannot. Uniqueness over an unpartitionable namespace and hard global limits genuinely require serialization.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Before paying for agreement, ask whether a violation can be repaired, whether the operations commute, and whether one node can own the decision. Most cases dissolve under one of these; some genuinely do not.
Practical
Partition by the key the invariant is scoped to, and coordinate only on ownership assignment. Where you accept repair-later, build and *test* the repair path, monitor the unrepaired delta rather than the job, and bound how far the system may drift before it starts refusing.
Advanced
The CALM theorem gives the exact boundary: a program has a coordination-free distributed implementation if and only if it is monotonic. Practically, look for the negation in the invariant statement — "no other", "not already", "does not exceed". That negation is where coordination is forced, and the design question becomes how narrowly you can scope it: per key rather than global, per epoch rather than per operation, per allocation rather than per resource.
Apply it
- 🔧 Take "inventory must never go negative" and produce three designs: fully coordinated, per-warehouse allocation, and record-then-evaluate. State each one’s guarantee and failure mode.
- 🔧 Find the negation in an invariant from your own system and describe the narrowest coordination that covers it.
- ⚡ A team wants a distributed lock around "apply a discount code, max 1000 uses". Propose an allocation-based design, and say precisely what it gives up.
- 💬 Give me three ways to satisfy an invariant without coordinating, and an example where none of them work.
- 💬 When is allowing a violation and repairing it better than preventing it?
- 💬 Why does partitioning ownership remove coordination, and where does the coordination go instead?
- 💬 What kind of invariant can never be made coordination-free?