The question this answers
What must never become false in my system, and what does that requirement force me to build?
An invariant is only protected within a stated scope. "Balance never negative per account, enforced at the single owner of that account" is a guarantee. "Balance never negative" is an aspiration until you say where it is checked, by whom, and what happens when that node is unreachable. This lesson guarantees nothing by itself; it is the discipline that makes every other guarantee in the module statable.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The decisive question for any invariant is: what must a node know to check it, and can it know that locally? An invariant over one account is checkable by whoever holds that account. An invariant over all accounts is not checkable by anyone without asking everyone. The distance between what the invariant requires and what a node can see locally is exactly the amount of coordination you are about to pay for.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Four invariants, four architectures
Take four requirements that sound similar in a product meeting and force entirely different systems. The difference is not the strength of the requirement — it is the scope over which it must hold and whether a violation can be undone.
Notice that none of these architectural conclusions mentions a technology. They are forced by the shape of the statement: what it ranges over, and whether it contains a negation over things a single node cannot see.
- "Balance must never go negative." Scope: one account. Checkable locally by the account’s owner. Architecture: partition by account, single owner per account, no global coordination. The strongest-sounding of the four needs the least machinery.
- "A username must be unique." Scope: the entire namespace, and it contains a negation ("no other user has it"). Not checkable locally by anyone. Architecture: a single decision point per name — route by hash of the name, or a database unique constraint. See Distributed Uniqueness: One Name, Many Shards.
- "At most one active lease owner." Scope: one resource, and the truth changes over time. Architecture: a lease with an expiry, plus a fencing token so a stale owner’s actions are rejected. See Leases: Authority With an Expiry Date and Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
- "An order cannot ship before payment is captured." Scope: two services, and the check is *ordering*, not exclusion. Architecture: no lock at all — make shipping consume an event that only exists after capture, so the invariant is structural rather than enforced. See Sagas: Trading Isolation for Availability.
How to state an invariant properly
Most invariants as first written are unbuildable, because they omit the three things that determine the architecture. A usable statement answers all three.
Run the fourth example through it: "an order cannot ship before payment is captured" scopes to one order; the enforcement point can be the shipping service reading the order’s state; a violation means goods left the building, which is expensive but not unrepairable — you can invoice or recall. That combination says: no distributed lock, one owner per order, and an event-shaped dependency. The architecture fell out of the statement.
- Scope — over what set must this hold? One row, one account, one tenant, everything? This decides whether it can be partitioned, and it is the single most consequential word in the sentence.
- Enforcement point — which component checks it, and what must that component be able to see at check time? If the answer is "everything", you have found your coordination requirement.
- Violation cost — what happens if it is briefly false? Unrepairable (a file overwritten, money gone), expensive but repairable (an oversell), or trivial (a duplicate email)? This decides prevention versus repair.
Deriving the architecture
With those three answered, the mechanism is close to determined. The mapping below is not a lookup table to apply mechanically, but it captures how much of the decision the invariant statement already makes.
The step teams skip is the first one. Starting from "we need a distributed lock" and working backwards produces a design nobody can evaluate, because there is no stated property to evaluate it against — and no way to know when the lock can be removed.
| Scope | Locally checkable? | Violation | Mechanism |
|---|---|---|---|
| Single key / entityprotocol | Yes, by the owner | Any | Partition + single owner; no coordination |
| Global uniquenessprotocol | No — negation over all | Poor repair | Route by key to one decider, or a unique constraint |
| Global limit / totalassumption | No | Repairable | Per-owner allocations, reconcile centrally |
| Global limit / totalassumption | No | Unrepairable | Serialize through one authority — pay the cost |
| Exclusive access over timeprotocol | No — depends on others | Unrepairable | Lease + fencing token at the resource |
| Ordering between two effectstypical | Yes, if the dependency is data | Any | Make the second consume the first’s output; no lock |
Invariants that are cheaper than they look
Two reframings recover a great deal of availability at no cost to correctness, and both are worth trying on any invariant that appears to demand global coordination.
Turn exclusion into ordering. "Only one process may do X" often really means "X must not happen twice concurrently in a way that corrupts state". If X can be made idempotent, or routed through a single-consumer queue, the exclusion requirement disappears and takes the lock with it — Work Queues: One Task, One Worker, Competing Consumers enforces single-handling structurally rather than by agreement.
Turn a global limit into local allocations. "No more than 1000 total" becomes "each of ten owners may issue at most 100", with periodic rebalancing. The global invariant is preserved exactly — the sum of local limits never exceeds the global one — while every decision becomes local. You lose some utilisation and gain complete independence, and for most rate limits and quotas that is a very good trade.
A third, less comfortable reframing: check whether the invariant is a business rule at all. "One active session per user" is often an assumption inherited from a single-server era, not a requirement anyone will defend when asked. Invariants that nobody will defend are the cheapest ones to remove.
Writing invariants down where they can be checked
An invariant that lives only in a design discussion decays. Six months later someone adds a code path that violates it, and nothing objects, because the property was never expressed anywhere a machine could evaluate.
The durable form is a continuous check: a query that returns the set of violating records, run on a schedule, alerting on a non-empty result. SELECT * FROM accounts WHERE balance < 0. SELECT username, COUNT(*) FROM users GROUP BY username HAVING COUNT(*) > 1. SELECT * FROM orders WHERE shipped_at < paid_at. These are cheap, and they catch violations from causes your coordination never anticipated — bad migrations, admin tooling, a bug in the owner itself.
This is also how the repair-later designs from Coordination Avoidance: Restructuring the Problem Instead of Paying for It become defensible. A repair path you cannot observe is a promise; a repair path with a violation query and a gap metric is a design.
Key points
- Name the invariant, with its scope, before choosing any mechanism.
- Scope decides everything: per-entity invariants partition and need no coordination; global ones with a negation do not.
- The gap between what the invariant requires and what a node can see locally is the coordination you must pay for.
- Violation cost decides prevention versus repair.
- Exclusion can often be reframed as ordering, and global limits as local allocations, with no loss of correctness.
- Write every invariant as a continuous violation query — otherwise it decays into folklore.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • State the invariant as a sentence that must never be false.
- • Identify its scope: the set of entities over which it ranges.
- • Identify the enforcement point and what that component must see to check it.
- • Classify the violation cost: unrepairable, expensive but repairable, or trivial.
- • If the scope partitions and each partition is locally checkable, assign single ownership and stop — you need no coordination.
- • If it does not partition, find the narrowest coordination that covers the negation, and amortise it into a lease or token.
- • Express the invariant as a query that finds violations, and run it continuously.
- • The invariant is stated without scope, so the design defends a stronger property than needed at a much higher cost.
- • The enforcement point cannot see what it needs at check time, so the check passes on stale data.
- • A second write path — an admin tool, a migration, a background job — bypasses the enforcement point entirely.
- • The invariant holds per node but not globally, and nobody notices because no global check exists.
- • The invariant changes as the product changes, and the coordination built for the old one remains, protecting nothing.
- • Bypass path: 99% of writes go through the guarded service and an admin script writes directly. The operator sees violations in the data with no corresponding entry in the service’s logs, and a coordination mechanism that appears to be working perfectly.
- • Stale-read check: the enforcement point reads a replica and checks against data that is 200 ms old. The operator sees violations clustering under high concurrency and none under load testing that lacks contention.
- • Over-broad coordination: a global lock protects a per-entity invariant. The operator sees serialised throughput on an endpoint that should scale horizontally, and lock wait time dominating latency.
- • Undetected drift: no continuous check exists, and a violation introduced by a deploy is discovered by a customer three weeks later. The operator sees a data-repair project instead of an alert.
- • Zombie invariant: coordination protects a rule the product removed two years ago. The operator sees an availability coupling that no one can justify and everyone is afraid to remove.
- • Coordination is required exactly where the invariant ranges over more than one node can see.
- • Narrowing the scope narrows the coordination; this is the highest-leverage design move available.
- • Amortising it — one agreement per lease or epoch — converts a per-request cost into a per-change cost without weakening the invariant.
- • A per-entity invariant survives a partition on the side that owns the entity, and the other side is simply unavailable for that entity.
- • A global invariant enforced by coordination survives partitions but is unavailable on the minority side.
- • A global invariant with no enforcement point survives nothing, and its violation is discovered later — usually by a customer.
- • Detect: the violation query, running continuously, alerting on a non-empty result.
- • Contain: close the bypass paths first — an unguarded write path defeats every mechanism upstream of it.
- • Recover: repair the violating records, and record how they arose.
- • Reconcile: for invariants spanning services, reconcile the two sides on a schedule; the disagreement set is the violation set. See Reconciliation Is a Component, Not a Cleanup Script.
- • Verify: after every incident, check whether the invariant statement itself was wrong — scope errors are more common than mechanism errors.
- • Violation count per invariant, as a graph. Zero is the expected value and any non-zero point is an event.
- • Time-to-detect: the gap between a violation appearing and the check noticing it.
- • Count of distinct write paths to each guarded entity — the bypass risk, and a number that only grows.
- • For repair-later invariants: the size and age of the unrepaired set.
- • At design time, before any mechanism has been chosen — this is where it pays for itself many times over.
- • When reviewing an existing design that "needs a distributed lock", to find out what property the lock is actually defending.
- • When deciding whether coordination can be *removed*, which is impossible to argue without a stated invariant.
- • When it becomes a documentation exercise producing invariants nobody checks — an unchecked invariant is worse than none, because it creates false confidence.
- • When stated too strongly out of caution, forcing global coordination for a property that only ever needed per-entity scope.
- • Enforce in a single database with a constraint: the cheapest correct enforcement point available, and correct far more often than distributed alternatives.
- • Structural enforcement: make the violating state unrepresentable — the second effect consumes the first’s output, so the ordering cannot be violated. No runtime check required.
- • Detect and repair rather than prevent, where the violation cost permits. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • Narrow the invariant with the product owner: many apparently global rules are acceptable per tenant or per region, which changes the architecture completely.
From invariant to architecture
SELECT name, count(*) FROM users GROUP BY name HAVING count(*) > 1
What people believe, and what is true
Stronger invariants are safer.
Stronger-than-needed invariants force coordination that reduces availability, and availability failures are also correctness failures from the user’s side.
The invariant is enforced because the service checks it.
It is enforced only if *every* write path checks it. Admin tools, migrations and background jobs are where invariants actually break.
We do not need a check because the code prevents it.
The code prevents it today. A continuous violation query is what tells you when that stops being true, and it catches causes the code never anticipated.
Every invariant needs a lock.
Most need an owner. Some need a constraint. A few need a lease and a fencing token. Locks are the mechanism of last resort, not of first choice.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Say precisely what must never be false and over what set. That statement, plus the cost of a violation, determines the architecture — usually before any technology is mentioned.
Practical
For each invariant write down scope, enforcement point and violation cost. Partition by the scope where you can, so the check becomes local. Close every bypass write path. Then express the invariant as a query that returns violations and alert on it being non-empty, so the property is checked by a machine rather than remembered by a person.
Advanced
The formal handle is that an invariant is coordination-free exactly when it is closed under the merge of any two states a node could independently reach — the I-confluence property. Per-entity constraints under single ownership are trivially I-confluent because no two nodes ever produce divergent states for the same entity. Uniqueness and hard global limits are not, and that is the precise reason they force serialization: two independently valid states merge into an invalid one.
Apply it
- 🔧 Write down three invariants from a system you work on with scope, enforcement point and violation cost, then derive the mechanism each one forces.
- 🔧 Find one invariant in your system with more than one write path and describe how it could be violated through the unguarded one.
- ⚡ A product owner says "a user can only have one active subscription". Establish the scope, the enforcement point and the violation cost, then propose an architecture — and say what you would build differently if the answer to violation cost were "we just refund them".
- 💬 Take "a username must be unique" and derive the architecture from it.
- 💬 Why does "balance must never go negative" need less coordination than "usernames must be unique"?
- 💬 How would you enforce "an order cannot ship before payment is captured" across two services?
- 💬 How do you know an invariant still holds in production today?