The question this answers
How do I decide which parts of my system actually need agreement?
An invariant is the guarantee. Everything else — a quorum, a lock, a leader, a transaction — is a mechanism whose only job is to keep some named predicate true. A system without stated invariants has no guarantee to evaluate, only mechanisms to defend.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows whether the invariant holds for the state it owns. It cannot know whether a system-wide invariant holds, because that would require observing all the relevant state simultaneously — which is exactly what a distributed system does not offer. So an invariant is enforceable only if it can be checked entirely within one owner, or if agreement is bought explicitly.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What an invariant actually is
An invariant is a predicate over system state that must be true at every externally-observable moment. "Account balance is never negative." "A seat is assigned to at most one booking." "Every shipped order has a captured payment." "Usernames are unique." Note what these have in common: each is a statement about *state*, is checkable by looking at data, and would be recognisable as violated by a person, not just by a test.
Contrast that with things that are often mistaken for invariants. "Writes are replicated within 100ms" is a service level objective — it can be missed without anything being *wrong*. "The order service calls the payment service" is a mechanism. "Data is consistent" is not a predicate at all; consistent with what? The discipline of writing invariants as falsifiable statements about data is what makes the rest of the design tractable.
The critical property, for this domain, is scope: how much state does the predicate range over? An invariant over a single row is enforceable by the database that holds the row, for free. An invariant over two rows in one database is enforceable by a transaction, cheaply. An invariant over state in two services is enforceable only by a distributed protocol, expensively. The scope of the invariant sets the price of the system, and this is the single most useful thing to know before drawing any boxes.
| Enforced by | Cost | What breaks it | |
|---|---|---|---|
| One rowprotocol | A column constraint or conditional update | Free | A second writer bypassing the check |
| Several rows, one storeprotocol | A local transaction | Cheap | Wrong isolation level |
| Several partitions, one systemassumption | Distributed transaction or a single-partition redesign | A round trip plus coupled availability | A participant failing mid-commit |
| Several servicesassumption | Saga with compensation, or accepted temporary violation | Intermediate states are externally visible | A compensation that cannot undo the effect |
| Several regionsprotocol | Cross-region consensus, or a per-region invariant instead | Tens of milliseconds per write; unavailable under partition | A region partition during a write |
Weakening the invariant is a design tool, not a defeat
When an invariant is expensive, the reflex is to build a bigger protocol. The more productive move is usually to ask whether the business needs the strong form. Airlines famously do not maintain "seats sold ≤ seats available" strictly; they overbook, detect the violation, and compensate with money. Banks do not maintain "balance ≥ 0" by holding a global lock; they allow the overdraft and charge for it. These are not sloppy engineering — they are recognitions that the cost of strict enforcement exceeded the cost of occasional violation plus a remedy.
The technique generalises into three moves. Localise it: partition the state so the invariant fits inside one owner — instead of a global counter, give each region a reserved allocation it can spend without asking. Bound it: allow violation within a limit, so a node can act independently up to a budget it holds. Detect and compensate: allow violation and pay for it, when the remedy is cheaper than the coordination.
What is *not* available is pretending. An invariant that the business genuinely requires, enforced by nothing, is not a weakened invariant — it is a defect waiting for the traffic that finds it. The value of writing invariants down is largely that it forces this to be a decision rather than an accident. Coordination Avoidance: Restructuring the Problem Instead of Paying for It and Start From the Invariant, Not From the Architecture are where the technique is developed.
- Localise: partition state so the predicate is checkable by a single owner.
- Bound: hand each node a budget it can spend without coordinating, and coordinate only to redistribute.
- Detect and compensate: allow the violation, notice it fast, and remedy it with a business action.
- Order-independent: choose operations that commute, so concurrent execution cannot violate the predicate.
- Not available: leaving a required invariant unenforced and calling it eventual consistency.
The two questions that price a design
For every invariant, two questions determine what the system must do. First: can it be violated temporarily? If yes, the system can be eventually consistent for that predicate, and you need a detection mechanism and a remedy. If no, every operation that could violate it must be serialised through something — an owner, a quorum, a lock.
Second: what is the cost of a violation? A duplicate notification costs a mild annoyance. A double charge costs a refund and trust. A double-assigned seat costs a customer. An oversold flight costs a voucher and is priced into the business model. The answers rarely line up with engineers’ instincts, which is why this question belongs to the product conversation and not to the design review.
Together these two answers give you the coordination map: the small set of operations that must be coordinated, and the large majority that must not be. Systems that skip this exercise tend toward one of two failure shapes — coordinating everything, and being slow and fragile; or coordinating nothing, and discovering their invariants through incidents.
Making invariants observable
An invariant you cannot check is an invariant you do not have. The practical form of this is a continuously-running verification: a query, a reconciliation job, or an assertion in the write path that tests the predicate against actual data and reports the number of violations. That number should be zero, and it should be on a dashboard, because this class of failure produces no errors and no exceptions.
This has a second benefit that is easy to miss. Once violations are measured, weakening an invariant becomes safe to do deliberately: you can allow bounded violation precisely because you can see how much occurs. Without the measurement, "eventually consistent" is indistinguishable from "occasionally wrong and nobody knows".
It also gives incident response a spine. The first question in a distributed incident is rarely "which service is erroring" — it is "which invariant is currently false, and how much state is affected". Teams that maintain a written list of system invariants answer that in minutes; teams that do not spend the incident discovering what the system was supposed to guarantee.
Key points
- An invariant is a falsifiable predicate over data that must never be observably false.
- The scope of an invariant — one row, one store, one service, many regions — sets the price of the whole system.
- Weakening an invariant deliberately (localise, bound, compensate) is a legitimate and often correct design move.
- Two questions price a design: may it be violated temporarily, and what does a violation cost?
- An invariant that is not continuously checked is not enforced, because this failure class is silent.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Write each invariant as a predicate over named state, in a sentence a non-engineer would recognise.
- • Determine its scope: the smallest set of state whose values decide whether it holds.
- • Place that state under a single owner if the scope allows it.
- • Where the scope cannot be localised, choose deliberately between coordination, bounded violation, and detect-and-compensate.
- • Implement a check that evaluates the predicate against real data on a schedule, and alert on nonzero violations.
- • A second writer appears and bypasses the enforcement path, so the constraint holds only for well-behaved callers.
- • The invariant is enforced by a read followed by a write, leaving a window where a concurrent operation slips through.
- • A partition makes the coordinating component unreachable, and the operation proceeds without the check.
- • A compensation fails or is impossible — an email cannot be unsent — so a temporary violation becomes permanent.
- • Silent drift: a cross-service invariant is violated and nothing errors. The operator sees a reconciliation count climbing and no corresponding alert on any service.
- • Enforcement by convention: the rule lives in one code path and a batch job bypasses it. The operator sees violations appearing in bursts on a schedule.
- • Check-then-act race: two concurrent requests both pass the check and both write. The operator sees two records that should have been mutually exclusive, created within milliseconds of each other.
- • Compensation gap: the invariant is restored in the database but the external effect has already been observed by a user. The operator sees a support ticket describing a state the system believes never existed.
- • An invariant whose scope fits inside one owner needs no distributed coordination — this is the whole reason partition keys are chosen the way they are.
- • An invariant spanning owners requires either serialising the relevant operations through a single point, or a consensus round per operation.
- • Bounded violation converts continuous coordination into occasional coordination: nodes spend from a local allocation and only coordinate to refill it, which is the essence of coordination avoidance.
- • Invariants local to a reachable owner continue to hold throughout any failure elsewhere.
- • Invariants spanning a failed boundary are unenforced for the duration, whether or not the system reports anything.
- • Bounded-violation designs continue to operate within their allocation and stop — rather than violate — when it is exhausted.
- • Detect: run the predicate as a query against real data; the count of violations is the signal.
- • Contain: stop the operations that can widen the violation before repairing the existing ones.
- • Recover: repair state to satisfy the predicate, using the owner as the authority.
- • Reconcile: for effects already visible externally, apply the business remedy rather than editing history.
- • Verify: re-run the check and confirm it returns to zero, and keep it running afterwards.
- • A violation count per invariant, evaluated against real data on a schedule, alerting on any nonzero value.
- • Rejection rate of the enforcement mechanism — conditional writes that failed, claims refused — which should be non-zero and stable.
- • Distinct write paths that touch the state an invariant ranges over; growth here is how enforcement quietly erodes.
- • Time from violation to detection, which is the real measure of how eventual your eventual consistency is.
- • At the start of any design, because it determines the topology rather than being determined by it.
- • During incidents, where "which invariant is false" is a faster route to the problem than "which service is unhealthy".
- • When a system feels slow because everything coordinates: the invariant list shows which coordination can be removed.
- • Enumerating invariants exhaustively for a prototype whose requirements are still moving is effort spent on a moving target.
- • Treating every business rule as an invariant produces a system that coordinates on things a nightly job could have fixed.
- • Enforce in the database with a constraint: unique indexes and check constraints are the cheapest, most reliable enforcement available, and they cannot be bypassed by a new code path.
- • Redesign the state so the invariant is local — one row, one owner — rather than building a protocol to span it.
- • Accept bounded violation with a reservation or budget model, coordinating only to redistribute the budget.
- • Accept violation and compensate, where the business remedy is cheaper than the coordination and the remedy actually exists.
Name the invariant, then price its scope
What people believe, and what is true
Strong consistency means my invariants are safe.
Linearizability orders operations; it does not know what your predicate is. An invariant spanning two linearizable stores is still unenforced.
Eventual consistency means the invariant will eventually hold.
It means replicas converge on values. If concurrent operations produced a state that violates the predicate, convergence preserves the violation faithfully.
We can enforce it in application code.
You can enforce it on the paths that run that code. A constraint in the store is enforced on all paths, including the ones written next year.
Weakening an invariant is a compromise we should avoid.
It is often the correct engineering decision, and it is only safe when it is deliberate and measured. The failure is weakening it by accident.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Write down what must never be false. The amount of state that predicate ranges over decides how expensive your system is. Shrink the scope before you build a protocol.
Practical
For each invariant: state it as a predicate, find its scope, put that state under one owner if you can, and add a check that runs against real data and reports violations. Where you cannot localise it, choose explicitly between coordinating, bounding the violation, and compensating — and write down which you chose and why.
Advanced
The formal treatment is invariant confluence: an invariant is enforceable without coordination exactly when every pair of states that individually satisfy it can be merged into a state that also satisfies it. Adding to a set is confluent — merge two sets and the result is still a set. Decrementing a counter with a floor of zero is not — two nodes each with balance 10 can each spend 10, and no merge rule recovers a valid state. This is a checkable property, and it explains cleanly why some operations replicate without agreement and others cannot: it is not about the data structure, it is about whether the predicate survives the merge.
Apply it
- 🔧 Write down the five most important invariants of a system you work on, and for each name the mechanism that enforces it and the check that would detect a violation. Expect at least one to have neither.
- ⚡ A ticketing system must not oversell, must scale to 50,000 concurrent buyers, and must stay available if one region fails. Work through the invariant scope and pick a design.
- ⚡ A reconciliation job reports 14 orders shipped without a captured payment. Walk through detection, containment and remedy.
- 💬 State an invariant from a system you have worked on, and say what enforces it.
- 💬 Why does the scope of an invariant determine the cost of the whole system?
- 💬 Give an example of an invariant that a business would happily let you weaken, and one it would not.