The question this answers
I have three copies of everything. Why did the outage still happen?
N copies guarantee survival of a failure only when at most N−1 of them are in the affected fault domain, the recovery path works without the failed component, and the remaining copies have the capacity to serve the whole load. Copies alone guarantee nothing.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows it is one of a set. It usually does not know where its peers are physically, what they share with it, or whether the set still constitutes a quorum. Placement is a property of the deployment, not something the software can verify at runtime — which is why placement drifts silently.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Copies are the cheap part
Adding replicas is easy: change a number, apply, watch three pods appear. Because it is easy, it is what gets done, and because the diagram then shows three boxes where there was one, it feels like the problem is solved. What actually determines whether the system survives is a set of things that are all harder and none of which the replica count expresses.
Placement. Three replicas that the scheduler happened to put on one node survive nothing that node does. Anti-affinity rules exist for exactly this and are frequently absent, or present as a preference rather than a requirement, which means they are silently ignored under capacity pressure. This is worth checking rather than assuming: the failure mode is that everything is correct on the day it was configured and drifts afterwards.
Independence. Covered at length in Correlated Failure: The Independence Assumption Is Usually False: replicas that share software, configuration and dependencies fail together, and the number of them is irrelevant to those causes.
Capacity. Three replicas each running at 60% do not survive the loss of one, because the survivors would need 90% each and they will not get there gracefully. Redundancy without headroom is redundancy that converts a single failure into a cascade.
A working recovery path. The failover has to actually happen: promotion has to complete, clients have to notice, connections have to be re-established, DNS or service discovery has to converge. Every one of those is code that runs rarely, and code that runs rarely does not work.
| Survives instance loss | Survives host loss | Survives zone loss | Survives bad deploy | |
|---|---|---|---|---|
| 3 replicas, 1 hostprotocol | Yes | No | No | No |
| 3 replicas, 3 hosts, 1 zoneprotocol | Yes | Yes | No | No |
| 3 replicas, 3 zonesprotocol | Yes | Yes | Yes | No |
| 3 replicas, 3 zones, progressive rollouttypical | Yes | Yes | Yes | Bounded to the rollout fraction |
| Any of the above at 90% utilisationassumption | Survives, then cascades | Cascades | Cascades | — |
The failover that has never run
The most reliable prediction in operations is that an untested recovery path will not work when it is first needed. The reasons are dull and consistent: the standby has been receiving replication but has never been promoted, so nobody knows how long promotion takes. The clients cache the primary’s address and do not re-resolve. The failover script references a host that was renamed. The standby’s configuration drifted. Credentials for the promotion path expired. The runbook is accurate for the previous version.
What makes this worse is that a failover attempt during an incident is the highest-stress, lowest-information moment to discover any of it — and a failed failover often leaves the system in a state worse than the original failure, with two half-promoted nodes or a split write path. This is why an untested failover is sometimes a *negative* asset: it invites an action under pressure whose outcome is unknown.
The remedy is to make it routine. If failover happens regularly — during planned maintenance, on a schedule, or as part of a chaos exercise — then its duration is known, its failure modes have been found, and the operator triggering it has done it before. Cloud infrastructure has the restore-testing analogue for backups, and the argument is identical: an untested backup is not a backup, and an untested failover is not redundancy.
replicas: 3/3 healthy ✓ all three responding placement: 3 pods ✗ all on node-7 (anti-affinity was "preferred") utilisation: 61% mean ✗ survivors would need 92% each failover: configured ✗ last exercised: never recovery deps: control plane ✗ in the same zone as the replicas Actual fault tolerance: one process crash.
Standby modes and what each really costs
Redundancy comes in a spectrum and the differences are mostly about the recovery path rather than the copies. Cold standby — a copy of the data and a plan. Recovery means provisioning, restoring and starting, measured in hours, and its main virtue is cost. Warm standby — running, receiving replication, not serving. Recovery means promotion and client redirection, measured in seconds to minutes, and the risk is that promotion is the part that has never been exercised. Hot / active-active — all copies serving. There is no failover step at all, which removes the untested-path problem entirely, and in exchange you take on the hardest version of the consistency question because writes can arrive anywhere.
Notice the trade being made: active-active is the most resilient precisely because its recovery path is its normal path. Nothing special happens when a node is lost; traffic simply stops going there. That is a general principle worth extracting — a recovery mechanism that is exercised continuously is the only kind you can rely on — and it applies well beyond replication. A retry path used constantly works; a retry path used once a year does not. A degraded mode that runs daily works; one that has never rendered does not.
Cloud infrastructure covers active-passive and active-active as deployment patterns, and the geo module here covers what active-active does to consistency across regions. The point for this lesson is the selection criterion: choose the mode whose recovery path you will actually exercise.
- Cold: cheapest, slowest, and the restore is the thing that must be tested.
- Warm: fast in theory; the promotion step is the untested part.
- Hot / active-active: no failover step at all, at the cost of the hardest consistency questions.
- Whichever you choose, the recovery path must run on a schedule, not on an incident.
Resilience is a property of the whole path
A last shift of frame. Redundancy is a property of a component; resilience is a property of the system’s response to a class of failure, and the difference shows up in what you measure. A redundancy metric is "how many replicas exist". A resilience metric is "when we removed one, what did users experience, and how long did it take to return to full redundancy".
The second is only obtainable by doing it. That is the case for fault injection and chaos engineering, and it is worth stating in its least dramatic form: the value is not in discovering exotic failures, it is in confirming that the ordinary ones behave as designed. Most such exercises find something mundane and important — a client that does not re-resolve DNS, a health check that lags reality by two minutes, a connection pool that never retries a dead endpoint.
It also reframes what to fix. If removing an instance causes a 30-second error spike, adding a fourth instance does not help; fixing the client’s connection handling does. Redundancy answers "what if a copy is lost". Resilience answers "and then what happens", and the second question is where user-visible impact actually lives.
Key points
- Replica count says nothing about placement, independence, capacity or recovery — all four decide whether the copies help.
- Anti-affinity as a preference rather than a requirement is silently ignored, and placement drifts.
- Redundancy without headroom converts one failure into a cascade.
- An untested failover is not redundancy, and can be worse than none because it invites a risky action under pressure.
- The recovery paths that work are the ones exercised continuously — which is the real argument for active-active.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Copies are created and kept in sync by a replication mechanism.
- • Placement decides which fault domains the copies occupy, and is controlled by the scheduler rather than by the replica count.
- • On failure, a detection mechanism decides a copy is gone — with all the caveats of No Heartbeat Does Not Mean Dead.
- • A recovery path promotes a survivor, redirects clients, and re-establishes connections.
- • The survivors absorb the failed copy’s load, which requires headroom that was reserved in advance.
- • The scheduler places all replicas in one domain because the constraint was advisory.
- • Promotion succeeds but clients continue using a cached address for the old primary.
- • Two nodes are promoted because detection was wrong, and both accept writes.
- • Survivors lack headroom and fail in sequence after the first loss.
- • The recovery path depends on a component in the same fault domain as the failure.
- • Co-located replicas: one host failure removes all copies. The operator sees three "independent" replicas disappear in the same second, and only then discovers the placement.
- • Failover that stalls: promotion begins and does not complete. The operator sees a cluster with no writable primary and a promotion job that has been running for minutes with no timeout.
- • Clients pinned to the dead primary: the database failed over correctly and the application did not notice. The operator sees a healthy new primary with no traffic and an application still erroring against the old address.
- • Cascade after a single loss: losing one instance pushes the rest past their limit. The operator sees failures at accelerating intervals and per-instance load climbing after each one.
- • Split write path: a failed failover leaves two nodes accepting writes. The operator sees divergent data and two nodes both reporting the primary role.
- • Promotion is a coordination decision and needs a quorum, or it is a race that can promote two.
- • Client redirection is a second, separate coordination problem, and it is the one most often left to caching and luck.
- • Active-active avoids the promotion coordination entirely and pays for it in write-path coordination instead — the cost moves rather than disappearing.
- • Copies in unaffected domains continue to hold correct data throughout.
- • The system is unavailable for writes from the moment of failure until promotion completes and clients converge — that whole window, not just the detection part.
- • Redundancy is reduced during recovery, so a second failure in that window is disproportionately damaging.
- • Detect: measure the whole failover, from failure to full client convergence, rather than the promotion step alone.
- • Contain: fence the old primary so a failed or slow failover cannot produce two writers.
- • Recover: restore full redundancy promptly, because the degraded window is where the second failure is expensive.
- • Reconcile: verify the returning replica against the current primary before it serves anything.
- • Verify: schedule the failover regularly so its duration is a known number rather than a hope.
- • Actual placement across fault domains, monitored continuously — the intended placement and the real one diverge silently.
- • Headroom per surviving replica: whether the remaining copies can absorb the load if one is lost right now.
- • Measured end-to-end failover duration from the last exercise, including client convergence, as a tracked number with a date.
- • Time spent at reduced redundancy after any failure, which is the window in which the system is far more fragile than its diagram suggests.
- • Whenever redundancy is being used as an availability argument in a design review or an audit response.
- • After any incident where redundant components failed together or a failover did not behave as expected.
- • For a stateless service that restarts in two seconds and loses nothing, elaborate failover machinery adds risk without adding availability.
- • Exercising failover on a system whose recovery path is known to be broken, without a plan, converts a latent problem into an incident at a time of your choosing — which is usually right, but should be deliberate.
- • Make recovery fast instead of failure rare: a stateless component that restarts in seconds may need no redundancy design at all.
- • Active-active, so the recovery path is the normal path and is exercised on every request.
- • Reduce blast radius so that a failure affects a fraction of users, rather than trying to make the failure survivable for all of them.
- • Accept the downtime: for many systems a documented, practised, twenty-minute recovery is cheaper and more reliable than automatic failover nobody has tested.
Three copies, and the outage still happened
What people believe, and what is true
We have three replicas, so we are fault tolerant.
You are tolerant of failures that affect at most two of them. Which failures those are depends on placement, and placement is not the replica count.
The failover is configured, so it will work.
Configured and exercised are different states. The first thing an unexercised failover reveals is how long it takes, and the answer is usually longer than anyone assumed.
Automatic failover is always better than manual.
Automatic failover acts on a suspicion that may be wrong. For rare, high-stakes transitions with a fast human response, manual with good fencing is a defensible choice.
Redundancy is what makes a system resilient.
Redundancy is a precondition. Resilience is what happens next — detection, promotion, client convergence, capacity absorption — and that is where user-visible impact lives.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Copies are cheap and prove nothing. What decides survival is where they are, what they share, whether the survivors have capacity, and whether the recovery path has ever run.
Practical
Monitor actual placement rather than intended placement. Make anti-affinity a requirement, not a preference. Size headroom so survivors can absorb the load plus the retry surge. Then schedule the failover — measure it end to end, including client convergence, and treat that number as the real availability input.
Advanced
The subtle cost is that redundancy adds failure modes of its own. Replication introduces lag, which introduces stale reads and the possibility of promoting a replica that is behind — losing acknowledged writes unless the protocol prevents it. Failover introduces the possibility of two primaries. More nodes mean more members whose membership must be agreed, and membership changes are a classic source of split brain when they overlap with a partition. So the honest evaluation is not "does redundancy improve availability" but "does the availability it adds exceed the availability its own machinery costs" — and for a component that restarts in two seconds and loses nothing, the answer is frequently no.
Apply it
- 🔧 Take a service you consider redundant and determine, from live data rather than configuration, which fault domains its replicas actually occupy and whether survivors have the headroom to absorb a loss.
- ⚡ A database fails over successfully in 8 seconds and the application is down for 6 minutes. Explain where the time went and what you would change.
- ⚡ Audit a three-replica deployment for real fault tolerance and list what you would measure continuously afterwards.
- 💬 Three replicas are running and healthy. What five things do you check before calling the service fault tolerant?
- 💬 Why can an untested failover be worse than no failover at all?
- 💬 Why is active-active more reliable in practice than warm standby, even ignoring capacity?