Coordination

Start From the Invariant, Not From the Architecture

Every coordination mechanism in this domain exists to protect some statement that must never become false. Name that statement first — precisely, with its scope — and the architecture follows from it. Skip that step and you will choose a mechanism, then reverse-engineer a justification.

▶ Run the lab

The question this answers

The question

What must never become false in my system, and what does that requirement force me to build?

The guarantee — the property claimed, and its scope

An invariant is only protected within a stated scope. "Balance never negative per account, enforced at the single owner of that account" is a guarantee. "Balance never negative" is an aspiration until you say where it is checked, by whom, and what happens when that node is unreachable. This lesson guarantees nothing by itself; it is the discipline that makes every other guarantee in the module statable.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

The decisive question for any invariant is: what must a node know to check it, and can it know that locally? An invariant over one account is checkable by whoever holds that account. An invariant over all accounts is not checkable by anyone without asking everyone. The distance between what the invariant requires and what a node can see locally is exactly the amount of coordination you are about to pay for.

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?What does a node know?How does it work?What can fail?How does it fail?Where is coordination?What holds under failure?How does it recover?How would you know?What is the simpler thing?
invariantsdesigncorrectnessscope

Four invariants, four architectures

Take four requirements that sound similar in a product meeting and force entirely different systems. The difference is not the strength of the requirement — it is the scope over which it must hold and whether a violation can be undone.

Notice that none of these architectural conclusions mentions a technology. They are forced by the shape of the statement: what it ranges over, and whether it contains a negation over things a single node cannot see.

  • "Balance must never go negative." Scope: one account. Checkable locally by the account’s owner. Architecture: partition by account, single owner per account, no global coordination. The strongest-sounding of the four needs the least machinery.
  • "A username must be unique." Scope: the entire namespace, and it contains a negation ("no other user has it"). Not checkable locally by anyone. Architecture: a single decision point per name — route by hash of the name, or a database unique constraint. See Distributed Uniqueness: One Name, Many Shards.
  • "At most one active lease owner." Scope: one resource, and the truth changes over time. Architecture: a lease with an expiry, plus a fencing token so a stale owner’s actions are rejected. See Leases: Authority With an Expiry Date and Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
  • "An order cannot ship before payment is captured." Scope: two services, and the check is *ordering*, not exclusion. Architecture: no lock at all — make shipping consume an event that only exists after capture, so the invariant is structural rather than enforced. See Sagas: Trading Isolation for Availability.

How to state an invariant properly

Most invariants as first written are unbuildable, because they omit the three things that determine the architecture. A usable statement answers all three.

Run the fourth example through it: "an order cannot ship before payment is captured" scopes to one order; the enforcement point can be the shipping service reading the order’s state; a violation means goods left the building, which is expensive but not unrepairable — you can invoice or recall. That combination says: no distributed lock, one owner per order, and an event-shaped dependency. The architecture fell out of the statement.

  • Scope — over what set must this hold? One row, one account, one tenant, everything? This decides whether it can be partitioned, and it is the single most consequential word in the sentence.
  • Enforcement point — which component checks it, and what must that component be able to see at check time? If the answer is "everything", you have found your coordination requirement.
  • Violation cost — what happens if it is briefly false? Unrepairable (a file overwritten, money gone), expensive but repairable (an oversell), or trivial (a duplicate email)? This decides prevention versus repair.

Deriving the architecture

With those three answered, the mechanism is close to determined. The mapping below is not a lookup table to apply mechanically, but it captures how much of the decision the invariant statement already makes.

The step teams skip is the first one. Starting from "we need a distributed lock" and working backwards produces a design nobody can evaluate, because there is no stated property to evaluate it against — and no way to know when the lock can be removed.

ScopeLocally checkable?ViolationMechanism
Single key / entityprotocolYes, by the ownerAnyPartition + single owner; no coordination
Global uniquenessprotocolNo — negation over allPoor repairRoute by key to one decider, or a unique constraint
Global limit / totalassumptionNoRepairablePer-owner allocations, reconcile centrally
Global limit / totalassumptionNoUnrepairableSerialize through one authority — pay the cost
Exclusive access over timeprotocolNo — depends on othersUnrepairableLease + fencing token at the resource
Ordering between two effectstypicalYes, if the dependency is dataAnyMake the second consume the first’s output; no lock
From the shape of the invariant to the mechanism

Invariants that are cheaper than they look

Two reframings recover a great deal of availability at no cost to correctness, and both are worth trying on any invariant that appears to demand global coordination.

Turn exclusion into ordering. "Only one process may do X" often really means "X must not happen twice concurrently in a way that corrupts state". If X can be made idempotent, or routed through a single-consumer queue, the exclusion requirement disappears and takes the lock with it — Work Queues: One Task, One Worker, Competing Consumers enforces single-handling structurally rather than by agreement.

Turn a global limit into local allocations. "No more than 1000 total" becomes "each of ten owners may issue at most 100", with periodic rebalancing. The global invariant is preserved exactly — the sum of local limits never exceeds the global one — while every decision becomes local. You lose some utilisation and gain complete independence, and for most rate limits and quotas that is a very good trade.

A third, less comfortable reframing: check whether the invariant is a business rule at all. "One active session per user" is often an assumption inherited from a single-server era, not a requirement anyone will defend when asked. Invariants that nobody will defend are the cheapest ones to remove.

Writing invariants down where they can be checked

An invariant that lives only in a design discussion decays. Six months later someone adds a code path that violates it, and nothing objects, because the property was never expressed anywhere a machine could evaluate.

The durable form is a continuous check: a query that returns the set of violating records, run on a schedule, alerting on a non-empty result. SELECT * FROM accounts WHERE balance < 0. SELECT username, COUNT(*) FROM users GROUP BY username HAVING COUNT(*) > 1. SELECT * FROM orders WHERE shipped_at < paid_at. These are cheap, and they catch violations from causes your coordination never anticipated — bad migrations, admin tooling, a bug in the owner itself.

This is also how the repair-later designs from Coordination Avoidance: Restructuring the Problem Instead of Paying for It become defensible. A repair path you cannot observe is a promise; a repair path with a violation query and a gap metric is a design.

Key points

  • Name the invariant, with its scope, before choosing any mechanism.
  • Scope decides everything: per-entity invariants partition and need no coordination; global ones with a negation do not.
  • The gap between what the invariant requires and what a node can see locally is the coordination you must pay for.
  • Violation cost decides prevention versus repair.
  • Exclusion can often be reframed as ordering, and global limits as local allocations, with no loss of correctness.
  • Write every invariant as a continuous violation query — otherwise it decays into folklore.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • State the invariant as a sentence that must never be false.
  • Identify its scope: the set of entities over which it ranges.
  • Identify the enforcement point and what that component must see to check it.
  • Classify the violation cost: unrepairable, expensive but repairable, or trivial.
  • If the scope partitions and each partition is locally checkable, assign single ownership and stop — you need no coordination.
  • If it does not partition, find the narrowest coordination that covers the negation, and amortise it into a lease or token.
  • Express the invariant as a query that finds violations, and run it continuously.
What can fail at the boundary
  • The invariant is stated without scope, so the design defends a stronger property than needed at a much higher cost.
  • The enforcement point cannot see what it needs at check time, so the check passes on stale data.
  • A second write path — an admin tool, a migration, a background job — bypasses the enforcement point entirely.
  • The invariant holds per node but not globally, and nobody notices because no global check exists.
  • The invariant changes as the product changes, and the coordination built for the old one remains, protecting nothing.
How it fails — what an operator sees
  • Bypass path: 99% of writes go through the guarded service and an admin script writes directly. The operator sees violations in the data with no corresponding entry in the service’s logs, and a coordination mechanism that appears to be working perfectly.
  • Stale-read check: the enforcement point reads a replica and checks against data that is 200 ms old. The operator sees violations clustering under high concurrency and none under load testing that lacks contention.
  • Over-broad coordination: a global lock protects a per-entity invariant. The operator sees serialised throughput on an endpoint that should scale horizontally, and lock wait time dominating latency.
  • Undetected drift: no continuous check exists, and a violation introduced by a deploy is discovered by a customer three weeks later. The operator sees a data-repair project instead of an alert.
  • Zombie invariant: coordination protects a rule the product removed two years ago. The operator sees an availability coupling that no one can justify and everyone is afraid to remove.
Where coordination is required
  • Coordination is required exactly where the invariant ranges over more than one node can see.
  • Narrowing the scope narrows the coordination; this is the highest-leverage design move available.
  • Amortising it — one agreement per lease or epoch — converts a per-request cost into a per-change cost without weakening the invariant.
What still holds under failure
  • A per-entity invariant survives a partition on the side that owns the entity, and the other side is simply unavailable for that entity.
  • A global invariant enforced by coordination survives partitions but is unavailable on the minority side.
  • A global invariant with no enforcement point survives nothing, and its violation is discovered later — usually by a customer.
How it recovers
  • Detect: the violation query, running continuously, alerting on a non-empty result.
  • Contain: close the bypass paths first — an unguarded write path defeats every mechanism upstream of it.
  • Recover: repair the violating records, and record how they arose.
  • Reconcile: for invariants spanning services, reconcile the two sides on a schedule; the disagreement set is the violation set. See Reconciliation Is a Component, Not a Cleanup Script.
  • Verify: after every incident, check whether the invariant statement itself was wrong — scope errors are more common than mechanism errors.
How you would know
  • Violation count per invariant, as a graph. Zero is the expected value and any non-zero point is an event.
  • Time-to-detect: the gap between a violation appearing and the check noticing it.
  • Count of distinct write paths to each guarded entity — the bypass risk, and a number that only grows.
  • For repair-later invariants: the size and age of the unrepaired set.
When it helps
  • At design time, before any mechanism has been chosen — this is where it pays for itself many times over.
  • When reviewing an existing design that "needs a distributed lock", to find out what property the lock is actually defending.
  • When deciding whether coordination can be *removed*, which is impossible to argue without a stated invariant.
When it hurts
  • When it becomes a documentation exercise producing invariants nobody checks — an unchecked invariant is worse than none, because it creates false confidence.
  • When stated too strongly out of caution, forcing global coordination for a property that only ever needed per-entity scope.
Simpler alternatives
  • Enforce in a single database with a constraint: the cheapest correct enforcement point available, and correct far more often than distributed alternatives.
  • Structural enforcement: make the violating state unrepresentable — the second effect consumes the first’s output, so the ordering cannot be violated. No runtime check required.
  • Detect and repair rather than prevent, where the violation cost permits. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
  • Narrow the invariant with the product owner: many apparently global rules are acceptable per tenant or per region, which changes the architecture completely.

From invariant to architecture

Start from the invariant, not from the architecture
Every coordination mechanism in this module exists to protect some statement that must never become false. Name that statement first — precisely, with its scope — and the architecture follows. Skip the step and you will choose a mechanism, then reverse-engineer a justification.
invariant
statement
A username must be unique
cost of a violation
Unrepairable — two people cannot share an identity, and picking a winner after the fact is a support incident.
what a node must know to check it
Every username in existence. This is a negation over global state, so no node can check it locally under any partitioning of the data.
the mechanism that forces
as stated
Partition the *decision*: route each name to exactly one owning shard by hash of the name, and let that shard do an atomic conditional insert. Coordination is confined to the ownership map and to rebalancing.
the check, as a query a machine runs continuously
SELECT name, count(*) FROM users GROUP BY name HAVING count(*) > 1
One write path, and it checks. Now count again in six months: this number only ever grows, which is why the continuous violation query matters more than the mechanism. An unchecked invariant is worse than none, because it creates false confidence.
The distance between what the invariant requires and what a node can see locally is exactly the amount of coordination you are about to pay for. Narrowing the scope is the highest-leverage design move available — and many apparently global rules turn out to be acceptable per tenant or per region, which changes the architecture completely.
protocolAn invariant containing a negation over global state ("no other X") cannot be checked locally by any node, regardless of implementation. That is a property of the statement, not of the system.
assumptionThe mapping from invariant shape to mechanism assumes the invariant is stated accurately. A scope error here produces a design that is either far too expensive or silently wrong, and no later mechanism choice can fix it.
typicalContinuous violation queries are cheap for small tables and need care at scale — sampling, incremental checks, or a replica with a stated staleness bound.

What people believe, and what is true

Claim

Stronger invariants are safer.

Reality

Stronger-than-needed invariants force coordination that reduces availability, and availability failures are also correctness failures from the user’s side.

Claim

The invariant is enforced because the service checks it.

Reality

It is enforced only if *every* write path checks it. Admin tools, migrations and background jobs are where invariants actually break.

Claim

We do not need a check because the code prevents it.

Reality

The code prevents it today. A continuous violation query is what tells you when that stops being true, and it catches causes the code never anticipated.

Claim

Every invariant needs a lock.

Reality

Most need an owner. Some need a constraint. A few need a lease and a fencing token. Locks are the mechanism of last resort, not of first choice.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

Say precisely what must never be false and over what set. That statement, plus the cost of a violation, determines the architecture — usually before any technology is mentioned.

Practical

For each invariant write down scope, enforcement point and violation cost. Partition by the scope where you can, so the check becomes local. Close every bypass write path. Then express the invariant as a query that returns violations and alert on it being non-empty, so the property is checked by a machine rather than remembered by a person.

Advanced

The formal handle is that an invariant is coordination-free exactly when it is closed under the merge of any two states a node could independently reach — the I-confluence property. Per-entity constraints under single ownership are trivially I-confluent because no two nodes ever produce divergent states for the same entity. Uniqueness and hard global limits are not, and that is the precise reason they force serialization: two independently valid states merge into an invalid one.

Apply it

Build it, then break it
  • 🔧 Write down three invariants from a system you work on with scope, enforcement point and violation cost, then derive the mechanism each one forces.
  • 🔧 Find one invariant in your system with more than one write path and describe how it could be violated through the unguarded one.
Reason about this
  • A product owner says "a user can only have one active subscription". Establish the scope, the enforcement point and the violation cost, then propose an architecture — and say what you would build differently if the answer to violation cost were "we just refund them".
Interview questions
  • 💬 Take "a username must be unique" and derive the architecture from it.
  • 💬 Why does "balance must never go negative" need less coordination than "usernames must be unique"?
  • 💬 How would you enforce "an order cannot ship before payment is captured" across two services?
  • 💬 How do you know an invariant still holds in production today?