The question this answers
What does it actually cost me to make two nodes agree before either acts?
A node that requires confirmation from a set S before acting is available for that action only when S is reachable. Its effective availability is the product of its own and S’s, not the minimum and certainly not its own. Nothing recovers that product; it can only be avoided by not requiring the confirmation.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node waiting for confirmation knows only that no confirmation has arrived yet. It cannot distinguish a peer that is dead, a peer that is slow, a network that dropped the request, and a network that dropped the reply. So "wait for agreement" is really "wait for an event that may never occur, with no way to tell which case you are in" — which is why every coordination point needs a timeout, and every timeout reintroduces A Timeout Tells You Nothing About Whether It Happened.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two prices, and the one nobody budgets for
The latency price is easy to see and easy to measure: one round trip to a quorum, added to every coordinated operation. Performance treats this well, and if latency were the whole story, coordination would be a tuning problem — put the nodes closer together, batch more aggressively, and move on.
The availability price is the one that changes architectures. Before coordination, node A serves requests whenever A is up. After, A serves requests only when A is up *and* it can reach the peers it must confirm with. A’s availability has become a function of the network and of machines A does not control. You have not added a dependency to a code path; you have added it to A’s definition of "working".
That is why this cost compounds so badly. Three independent services at 99.9% each, called in sequence, give roughly 99.7%. Three services that must all *agree* before any of them acts give something worse, because now the network between them is in the product too, and network partitions are exactly the events that are correlated across all three.
| A’s obligation before acting | A during a partition | What the user sees |
|---|---|---|
| None — acts locallyprotocol | Keeps serving | Full service, possibly divergent state |
| Read a cached authority (bounded stale)typical | Keeps serving until the bound expires | Full service, then degraded |
| Confirm with a majorityprotocol | Serves only on the majority side | Total failure for the minority |
| Confirm with *all* peersprotocol | Stops on any single peer loss | Failure whenever anything is down |
The coupling is the point, not a side effect
It is tempting to read this as a flaw to be engineered around. It is not. Coordination *is* the deliberate coupling of availability in exchange for a guarantee, and if you remove the coupling you have removed the guarantee. A node that acts without confirming has, by definition, acted without knowing what its peers did.
So the design question is never "how do I get agreement without the cost?" — there is no such thing. It is "which operations genuinely need this coupling, and how few of them can I get away with?" Every operation inside a coordination boundary inherits the availability of the whole boundary; every operation outside it keeps its own.
This is also the honest reading of CAP: What the Theorem Actually Says. CAP is often taught as a menu you choose from once. It is better understood as this trade made per operation: each operation either requires coordination and therefore stops during a partition, or does not and therefore may diverge. A real system makes that choice dozens of times, and the interesting engineering is in making the choice deliberately rather than by accident.
Where the cost hides
Coordination points are rarely labelled as such. They appear as ordinary infrastructure, and the availability coupling arrives with them unannounced. A useful exercise is to walk a request path and mark every point where the handler waits for a machine it does not control before it can proceed.
The last two are the ones teams miss, because they do not look like agreement — but a synchronous call that must succeed before you may act is exactly the same coupling as a quorum write, with a quorum of one and no majority argument to protect you.
- A distributed lock taken before the work — your availability is now the lock service’s. See Distributed Locks: What They Are Actually For.
- A synchronous quorum write — you are available only on the majority side.
- A uniqueness check against a shared store before insert — see Distributed Uniqueness: One Name, Many Shards.
- Two-phase commit across services — every participant can block every other. See The Blocking Window: When 2PC Stops and Waits.
- A synchronous config or feature-flag lookup on the request path, where a cached value would have been correct.
- A synchronous authorization or entitlement call that must return before the operation proceeds.
Making the trade visible
The practical technique is to give every coordination point an explicit degradation policy, decided in advance rather than during an incident. When the peers are unreachable, does this operation fail closed, fail open, or proceed on stale information with a bounded staleness?
Writing that down forces the real conversation. "Fail closed" is correct for a balance check and absurd for a feature flag. "Proceed on stale data" is correct for a config value and dangerous for a lock. The policy is where the CAP: What the Theorem Actually Says choice actually gets made, and having it in code rather than in someone’s head is the difference between a graceful degradation and a full outage.
The complementary move is to reduce the number of points at all — which is what Coordination Avoidance: Restructuring the Problem Instead of Paying for It is about, and why Start From the Invariant, Not From the Architecture insists you name the invariant before designing the mechanism.
Key points
- Coordination costs latency, but its structural cost is coupled availability.
- A node that must confirm before acting is unavailable exactly when its peers are unreachable.
- Effective availability is a product, and partitions correlate the terms — the compounding is worse than independent failures suggest.
- The coupling is the guarantee; removing it removes the property you were buying.
- CAP is a per-operation trade, not a one-time system-level choice.
- Every coordination point needs an explicit degradation policy decided before the incident.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • An operation is classified as requiring agreement with a set of peers.
- • The handler sends a request to those peers and blocks.
- • It proceeds only when enough replies arrive to satisfy the rule — all, a majority, or one specific authority.
- • If replies do not arrive within the deadline, the handler cannot distinguish failure from slowness and must apply its degradation policy.
- • That policy — fail closed, fail open, or act on stale information — determines what the user experiences during every partition thereafter.
- • The confirmation request is lost, so the operation blocks on an event that will never happen.
- • The confirmation reply is lost, so the peers acted but the initiator does not know.
- • A peer is slow rather than down, so the timeout fires while the operation is still live on the other side.
- • Enough peers are unreachable that no quorum forms and the operation cannot proceed at all.
- • Coordination points are nested, so one slow dependency stalls threads holding another coordination resource.
- • Total unavailability from partial failure: one dependency in a coordination set is down and every coordinated operation fails. The operator sees healthy application servers, a healthy database, and near-100% errors on one endpoint family.
- • Thread-pool exhaustion: coordinated calls block waiting for peers, threads accumulate, and endpoints with no coordination at all start failing too. The operator sees latency rising on unrelated routes — the coupling has escaped its blast radius.
- • Silent unavailability window: the coordination point has a long timeout, so requests do not error, they hang. The operator sees a latency cliff and client-side timeouts rather than server-side errors, and dashboards that look healthy.
- • Correlated degradation across services: three services coordinate, one network segment degrades, and all three lose availability simultaneously because the network is in every product term. The operator sees a fleet-wide incident from a single-segment fault.
- • Cross-region coupling: a coordination point acquires a peer in another region after a routine failover, and p99 for every coordinated operation steps up permanently with no code change.
- • This lesson is about the cost itself, so the coordination content is the point: every synchronous confirmation is a shared fate you have chosen.
- • The cost is paid per operation unless the agreement is amortised — a lease, a token, a cached authority with a bound.
- • Amortisation is the only real lever: the same guarantee, purchased once and reused, rather than purchased per request.
- • Operations inside the coordination boundary stop; the guarantee they protect is preserved.
- • Operations outside it continue and may diverge.
- • The system’s user-visible behaviour during a partition is decided entirely by which operations were placed inside.
- • Detect: measure the fraction of requests that synchronously wait on a machine you do not control, per endpoint.
- • Contain: bound every wait, isolate coordinated calls in their own thread pool or concurrency limit, and never let a coordinated call hold a resource an uncoordinated one needs.
- • Recover: restore peer reachability; coordinated operations resume without intervention.
- • Reconcile: for operations that proceeded on stale information, run the reconciliation that the degradation policy promised.
- • Verify: game-day each coordination point by making its peers unreachable and confirming the observed behaviour matches the written policy.
- • Per-endpoint count of synchronous coordination waits, and their p99 duration.
- • Availability of each endpoint conditioned on each dependency being reachable — the product made visible.
- • Timeout rate at each coordination point, separated from error rate.
- • Number of distinct coordination points on the critical path of the top user journeys; this number should be small and known.
- • When an invariant genuinely spans nodes and a violation cannot be repaired — the coupling is what you are buying.
- • When the cost is amortised: agreement once per lease, per epoch or per ownership change, reused across many operations.
- • When the coordinating peers share a failure domain anyway, so the coupling adds little real correlation.
- • On high-frequency operations where the guarantee is not required by any stated invariant.
- • Across failure domains — regions, availability zones, third parties — where the coupling introduces correlation that did not previously exist.
- • When the number of coordination points on a path grows past two or three, at which point availability arithmetic dominates every other consideration.
- • Partition ownership so the decision is local to one node and no confirmation is needed. See Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • Amortise: acquire a lease or token once and act independently until it expires. See Leases: Authority With an Expiry Date and Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely.
- • Act now, reconcile later, where the invariant tolerates a repair window. See Reconciliation Is a Component, Not a Cleanup Script.
- • Cache the authoritative value with a bounded staleness, converting a hard dependency into a soft one with a known window.
- • Move the coordination off the request path entirely — decide asynchronously and let the user-facing operation proceed optimistically.
Coordination couples availability
What people believe, and what is true
Coordination costs latency; we can absorb a few milliseconds.
The latency is the visible half. The other half is that the operation now fails whenever its peers are unreachable — a step change in behaviour, not a slowdown.
A quorum makes the system more available.
It makes it more available than requiring *all* peers, and strictly less available than requiring none. It is a compromise within a cost, not an escape from one.
Adding a lock is cheap because the lock service is fast.
Speed is irrelevant to the coupling. Your operation is now unavailable whenever the lock service is unreachable, however fast it is when it is up.
Timeouts protect us from the coupling.
A timeout converts a hang into an error. It removes the resource exhaustion, not the unavailability — and it reintroduces ambiguity about whether the peer acted.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Making nodes agree before acting couples their availability. A node that must confirm cannot act when its peers are unreachable, so every coordination point narrows when your system works.
Practical
Count the synchronous coordination points on each critical path and keep the number small. Give each one a written degradation policy — fail closed, fail open, or bounded-stale — and game-day it. Isolate coordinated calls so their waits cannot exhaust resources that uncoordinated paths need.
Advanced
Model each coordination point as multiplying availability by the reachability of its quorum, then note that partitions correlate the terms, so the product understates the damage. The only lever that changes the shape rather than the constant is amortisation: converting per-operation agreement into per-epoch agreement plus a locally checkable token. That is why leases and fencing exist, and why they appear in every mature distributed design.
Apply it
- 🔧 Map one critical user journey and mark every synchronous coordination point. For each, write the degradation policy and then test it.
- 🔧 Take an operation that currently coordinates per request and redesign it to coordinate per epoch instead. State what new mechanism the correctness now depends on.
- 💬 Beyond latency, what does it cost to require agreement before acting?
- 💬 You add a distributed lock to an endpoint. What has changed about that endpoint’s availability?
- 💬 Why does adding coordination points compound worse than a simple product of availabilities suggests?