The question this answers
This decision feels like it needs agreement. Does it actually need consensus?
Nothing is guaranteed by this lesson — it is a decision procedure. What it establishes is the scope rule: consensus should govern the facts that authorise work, not the work itself. A cluster running consensus over its configuration and leadership, with business operations flowing through the authority that grants, keeps the guarantee and pays for it once.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The question to ask of every candidate decision is what a node must know before acting, and how stale that knowledge may be. If a node can act correctly on a fact that is seconds old, it does not need consensus for that fact — it needs a cache with a bounded staleness. Only when acting on stale information is *incorrect*, rather than merely suboptimal, does agreement become mandatory.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The four legitimate uses
There is a short list of things consensus is genuinely for. What they share is that they are low-frequency facts that authorise high-frequency work, and that being wrong about them corrupts everything downstream.
Note the ratio in each case: one decision per hours or days, authorising millions of operations. That ratio is what makes the round-trip cost invisible. When the ratio approaches one-to-one, you are using consensus wrongly.
- Replicated metadata — shard maps, topology, feature-flag authority, schema versions. Small, rarely changed, and catastrophic if two nodes disagree.
- Single active leader — exactly one writer, one scheduler, one compactor. The classic case, and the reason Leader Election: Choosing One, and Knowing You Were Chosen exists.
- Configuration authority — the one place that says what the current configuration *is*, so a rollout does not produce two truths.
- Membership — who is in the cluster. Every quorum calculation depends on agreement about the denominator, so this one is self-referentially required. See Cluster Membership: A Belief, Not a Fact.
The anti-pattern: consensus on the request path
The seductive mistake is to notice that consensus gives strong guarantees and to route business operations through it. Every order, every message, every write becomes a log entry in a Raft group. It works beautifully in testing and then imposes two hard ceilings that no amount of tuning removes.
A throughput ceiling, because every operation is serialised through one leader and must reach a majority’s disks. And an availability floor, because every operation now requires a majority — so a system that could have degraded gracefully to "reads only" or "eventually consistent" instead stops completely. You have coupled the availability of every business operation to the availability of a quorum, which is precisely the coupling Coordination Couples Availability is about.
The tell is in the metrics: p99 latency that tracks your slowest quorum member, and an outage profile where partial infrastructure failure produces total application failure. If losing two of five nodes takes down checkout, checkout is inside the consensus group and probably should not be.
| Decision | Frequency | Cost of being wrong | Needs consensus? |
|---|---|---|---|
| Who owns shard 7protocol | Rare | Two writers, corruption | Yes |
| Current cluster membershipprotocol | Rare | Two disjoint quorums | Yes |
| Is this username takentypical | Per signup | Duplicate identity | No — route by key; see [[distributed-uniqueness]] |
| Order total for this carttypical | Per request | Recompute and correct | No — single owner per cart |
| Has this email been senttypical | Per message | Duplicate email | No — dedup key, idempotence |
| Account balance never negativeassumption | Per transaction | Real money lost | Only if it cannot be partitioned by account |
The questions to ask, in order
Run a candidate decision through these five questions. Most decisions fall out at the first or second.
If you reach the fifth question and the answer is still yes, use consensus, and put it where it authorises rather than where it executes.
- 1. Can the decision be made by one owner? If the data can be partitioned so exactly one node owns each key, that node decides alone. Ownership assignment needs consensus; the decisions themselves do not. This dissolves more cases than any other move.
- 2. Can a violation be detected and repaired later? Overbooking a flight, an occasional duplicate notification, a briefly negative counter — if the business already has a repair path, agreement up front is buying something it does not need. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- 3. Do the operations commute? Adding to a set, incrementing a counter, merging edits — if order does not affect the outcome, there is nothing to agree on. See CRDTs: Deterministic Merge, Not Correct Merge.
- 4. Is stale-but-bounded good enough? Reading a config that is up to five seconds old is usually fine. A cached lease with a known bound gives you almost all of consensus’s value at a fraction of its cost.
- 5. Does an invariant span multiple nodes and admit no repair? Only here is consensus mandatory — and even then, restructuring the invariant is worth one more attempt. See Start From the Invariant, Not From the Architecture.
Consensus as an authority, not as a pipe
The pattern that works is a strict separation. A small consensus group holds a small amount of slow-changing state: who owns what, what the configuration is, who is a member. Everything else runs on nodes that hold a *lease* or a *token* issued by that group and then act independently at full speed.
This is why etcd runs the Kubernetes control plane and not its data plane; why a database with a Raft-managed shard map handles millions of queries per second; why a lock service issues a lease and then gets out of the way. The consensus group is consulted when authority changes, not when work is done.
The safety of that separation rests entirely on Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely. Acting on a lease you may no longer hold is safe only when the resource rejects stale tokens. Without that check, the fast path is fast and wrong, and you have all of consensus’s cost with none of its protection — the worst of the available positions.
Key points
- Consensus belongs on facts that authorise work, not on the work itself.
- The legitimate uses are replicated metadata, single active leader, configuration authority and membership.
- The ratio that makes it affordable is one decision authorising millions of operations.
- Consensus on the request path imposes a throughput ceiling and an availability floor that tuning cannot remove.
- Most candidate decisions dissolve under "can one owner decide?" or "can a violation be repaired later?".
- Authority-plus-lease is only safe when the resource checks fencing tokens.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Identify the decision and state the invariant it protects, precisely.
- • Ask whether the state can be partitioned so a single owner decides — if yes, use consensus only to assign ownership.
- • Ask whether a violation is detectable and repairable — if yes, prefer detection and repair.
- • Ask whether the operations commute — if yes, prefer convergent structures.
- • Ask whether bounded staleness suffices — if yes, prefer a cached lease.
- • If none apply, place a small consensus group over the authorising fact and issue leases with fencing tokens for the work.
- • The consensus group becomes a dependency of every request, so its availability becomes the system’s availability.
- • Lease holders act on expired leases against unfenced resources.
- • The metadata under consensus grows until the group is handling a data-sized workload it was never sized for.
- • A "small" consensus group is placed across regions, making every authority change cost an inter-region round trip.
- • Ownership assignment churns, so the fast path is constantly interrupted by authority changes.
- • Total outage from partial failure: two of five nodes are lost and the entire application stops, because business writes go through the consensus log. The operator sees healthy application servers, a healthy database, and 100% write errors.
- • Throughput ceiling that scaling does not lift: adding application servers changes nothing because every write serialises through one leader. The operator sees flat throughput, rising queue depth, and a single node at high disk utilisation.
- • Metadata store used as a database: the coordination service degrades under a load it was never designed for, taking leadership and configuration down with it. The operator sees etcd or ZooKeeper latency spikes correlating with application traffic volume rather than with deploys.
- • Silent double-execution: an authority-plus-lease design where the resource never checks the token. The operator sees duplicate side effects during every failover, with no errors in any log.
- • Cross-region authority thrash: the consensus group spans regions and leadership migrates on ordinary latency variation. The operator sees write latency oscillating between 5 ms and 120 ms with no code or traffic change.
- • The design goal is to pay for coordination once per authority change rather than once per operation.
- • Whatever coordination remains should be on the slowest-changing facts, because that is where its cost is amortised best.
- • Every operation that consults the consensus group synchronously inherits its availability — that inheritance is the thing to count, not the milliseconds.
- • A well-scoped design degrades: authority changes stop, existing lease holders keep working until their leases expire, and the system loses the ability to *change* rather than the ability to *operate*.
- • A badly-scoped design stops entirely, because every operation needed the quorum.
- • The difference between those two outcomes is a scoping decision made long before the incident.
- • Detect: measure the fraction of requests that synchronously touch the consensus group — the single best predictor of your outage profile.
- • Contain: give lease holders a defined behaviour for "my lease expired and I cannot renew" — usually stop, sometimes continue read-only, never continue writing.
- • Recover: restore the quorum; lease holders re-acquire and resume.
- • Reconcile: audit for effects emitted on expired leases, which is where unfenced resources show up.
- • Verify: run a game day that removes a minority of the consensus group and confirm the business path still serves.
- • Percentage of user-facing requests that touch the consensus group synchronously.
- • Consensus decision rate — if it tracks user traffic rather than deploys and failovers, the scoping is wrong.
- • Lease renewal failures and their outcome at the holder.
- • Size and growth of the state under consensus, which should be near-flat.
- • Blast-radius test result: with a minority of the group down, what fraction of user journeys still complete?
- • When exactly one actor must own a resource and being wrong corrupts it.
- • When configuration or membership must be unambiguous across the fleet.
- • When automatic failover must be correct without a human adjudicating which replica was ahead.
- • On the request path for ordinary business operations.
- • Across regions for anything latency-sensitive.
- • For invariants that could be partitioned by key, where a single owner would have sufficed.
- • For invariants the business already repairs after the fact — you are paying to prevent something it is happy to fix.
- • Partition by key so one node owns each decision; use consensus only to assign ownership. The highest-leverage move available.
- • Detect and repair: allow the violation, find it with reconciliation, and compensate. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It and Reconciliation Is a Component, Not a Cleanup Script.
- • Commutative operations and convergent data types, where order does not matter — see CRDTs: Deterministic Merge, Not Correct Merge.
- • A single relational database with a uniqueness constraint or a transaction: a genuine, well-understood coordination point that most teams already operate well. Far simpler than a consensus group and correct for a great many "we need agreement" problems.
- • Bounded-staleness caching of an authoritative value, which removes the synchronous dependency at the cost of a known staleness window.
Do you actually need consensus?
| Consensus scoped to authority | Consensus in the request path | |
|---|---|---|
| On losing a minority of the groupprotocol | Authority changes stop; existing lease holders keep working until their leases expire. The system loses the ability to *change*, not the ability to *operate*. | Everything stops. Healthy application servers, a healthy database, and 100% write errors. |
| Throughputtypical | Bounded by the data plane. Adding servers adds throughput. | Bounded by one leader’s disk. Adding servers changes nothing. |
| Cross-region costprotocol | One inter-region round trip per authority change — rare. | One per request, with a latency floor set by the speed of light. |
| What to measuretypical | Consensus decision rate: it should track deploys and failovers, not user traffic. | If the decision rate tracks user traffic, the scoping is wrong. |
What people believe, and what is true
Consensus makes the system more reliable.
It makes decisions unambiguous. It makes availability *worse* under partition, deliberately, and anything placed inside it inherits that.
If we need strong consistency, we need consensus everywhere.
Strong consistency is needed for specific invariants, not for every operation. Most systems need it for a small subset and pay for it everywhere.
A distributed lock is a lightweight alternative to consensus.
A correct distributed lock *is* consensus, usually rented from etcd or ZooKeeper. It is a different interface, not a cheaper guarantee.
We will start with consensus everywhere and optimise later.
It sets your availability floor from day one, and moving operations out of a consensus log later means changing the guarantee they were built on — the hardest kind of migration.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Use consensus for the few facts that authorise work — leadership, ownership, configuration, membership. Keep it out of the path of ordinary operations, where it caps throughput and floors availability.
Practical
Run every candidate decision through: can one owner decide it? can a violation be repaired? do the operations commute? is bounded staleness enough? Only if all four fail does consensus become mandatory. Then measure the fraction of user requests that synchronously touch the consensus group, and game-day a minority failure to see whether your business path survives.
Advanced
The scoping question is really about where availability is *coupled*. A node that must reach agreement before acting has its availability multiplied by the quorum’s availability, and multiplication of probabilities below one is unforgiving. Restricting coordination to authority changes means user-facing availability depends on the quorum only at lease-renewal boundaries — turning a multiplication into a much rarer dependency, and turning a total outage into a bounded degradation window.
Apply it
- 🔧 Take a system you work on, list every decision that requires agreement, and classify each with the five questions.
- 🔧 Design the game day that proves your business path survives losing a minority of the consensus group.
- ⚡ A team wants strict global ordering of all user events "for auditability". Work out what they actually need, and propose a design that achieves it without a global consensus log.
- 💬 When should a system use consensus, and when is it the wrong tool?
- 💬 A team proposes putting every order through a Raft log. What do you tell them, and what evidence do you ask for?
- 💬 Your service needs "exactly one worker per shard". Does that need consensus? Walk me through the reasoning.
- 💬 What is the difference between using consensus as an authority and using it as a pipe?