The question this answers
When this thing fails, what else goes with it?
Redundancy guarantees survival of a failure only if the redundant copies are in different fault domains for that failure. Two instances in the same domain survive nothing that domain-wide failures cause, regardless of how many there are.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows what it can reach right now. It does not know why something became unreachable, and therefore cannot tell a single-instance failure from a domain-wide one — which matters, because the correct response differs: retry a peer for the first, shed load and degrade for the second.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The visible hierarchy, and the invisible one
The hierarchy everybody knows runs: process — a crash, an OOM kill, a deploy; machine or VM — hardware failure, a host reboot, live migration; rack — a top-of-rack switch, a power distribution unit; availability zone — a datacentre building, its power, its cooling, its network; region — a metro area, a natural disaster, a regional control-plane failure. Each level contains the ones below it, and infrastructure providers expose the middle levels explicitly so you can place things across them.
The hierarchy that causes outages is the invisible one. Shared dependencies cut across the physical layout and take down things in different zones simultaneously: a configuration service, a secrets manager, a container registry, an identity provider, a DNS zone, a certificate authority. Shared control planes are worse, because they are often not on the architecture diagram at all — the thing that starts new instances, the thing that updates routing, the thing that issues credentials.
And then there is the domain that spans everything: the deploy pipeline. A bad change reaches every zone and every region because that is what a deploy is for. Your carefully multi-zone, multi-region system has a single point of failure named "the last release", and it is the most common cause of large outages by a wide margin. That is a fault domain, and it is the argument for staged rollouts — cloud infrastructure has the canary and blue/green treatment, and the systems point is simply that a deploy defeats physical redundancy by design.
- config service (zone A only) — not on the architecture diagram
- b1believes “it is protected because it is in a different zone from A”✕ and it is false
- c1believes “the outage is a zone A problem”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Placement is the mechanism
Once you know the domains, the design act is placement: distribute the copies of anything you care about across domains at the level you intend to survive. Surviving a machine failure means two copies on different machines. Surviving a zone failure means copies in different zones — and specifically, it means that a majority of any quorum must survive the loss of one zone, which is why three zones behave completely differently from two.
The two-zone case is worth working through because it is a common and expensive mistake. With a three-node quorum split two-and-one across two zones, losing the zone with two nodes leaves one node, which is not a majority, and the system stops accepting writes. Losing the other zone leaves two, which is. So a two-zone deployment survives one of its two possible zone failures and not the other — a 50% chance of an outage that the architecture diagram claims is covered. Three zones with one node each survives any single zone loss, which is why quorum systems are almost always deployed across three.
The same reasoning applies to any redundancy: count what survives, do not count what exists. Six instances across two zones is six instances and two fault domains. The number that matters for zone failures is two.
| Survives 1 machine | Survives 1 zone | Survives 1 region | |
|---|---|---|---|
| 5 nodes, 1 zoneprotocol | Yes (up to 2) | No — total loss | No |
| 3 + 2 across 2 zonesprotocol | Yes (up to 2) | Only if the 2-node zone fails | No |
| 2 + 2 + 1 across 3 zonesprotocol | Yes (up to 2) | Yes, any zone | No |
| 3 regions, cross-region quorumprotocol | Yes | Yes | Yes — at tens of ms per write |
Finding the domains you did not know about
The reliable method is to ask, for each component, what must be working for this to start, and what must be working for this to keep running. The two answers differ, and the first one is where the surprises live. A service may run happily without its configuration store and be unable to start without it — which means a simultaneous restart of the fleet during a config-store outage turns a degraded state into a total one. This is the mechanism behind a large fraction of long outages: the system survived the initial fault and could not recover because recovery had a dependency the steady state did not.
A second method is to look at what is *not* replicated. Everything in a system tends to acquire redundancy except the things nobody thought of as infrastructure: a single build agent, one person’s credentials, a DNS record with one authoritative provider, a certificate that expires, a single queue that all failover traffic must pass through. Each is a fault domain containing everything.
A third is to examine correlations in past incidents. If two services that "share nothing" have gone down together twice, they share something, and finding out what is more valuable than any amount of diagramming.
- For each component: what must be up for it to *start*, versus to *keep running*?
- What is deployed to every zone at once, and what gates that?
- Which single instances exist that nobody calls infrastructure?
- What do the failover paths depend on that the steady-state paths do not?
- Which pairs of "unrelated" services have failed together before?
Blast radius is a design parameter
The other half of fault-domain thinking is deliberately *limiting* what a single failure can affect. Cell-based architectures divide the fleet into independent slices, each with its own copy of the stack and its own subset of users, so that a failure — including a bad deploy, a poison request, or a hot tenant — affects one cell rather than everyone. The cost is duplicated infrastructure and a routing layer that must itself be highly available; the benefit is that the worst case becomes a fraction rather than the whole.
Shuffle sharding is the same idea applied more cheaply: assign each tenant a random subset of the available workers rather than all of them, so a tenant that poisons its workers takes down only the tenants that happen to share the same subset. With modest numbers this makes complete overlap between any two tenants extremely unlikely, and it costs nothing but a routing rule.
Both are answers to the same question — how much of the system may one failure reach? — and both are worth considering before adding another nine to a component’s availability, because reducing blast radius is usually cheaper than raising reliability and it works against causes you did not anticipate.
Key points
- A fault domain is what fails together; the physical hierarchy is the easy half.
- Shared dependencies and control planes cut across zones and are usually missing from the diagram.
- The deploy pipeline is a fault domain spanning everything, by design.
- Count what survives, not what exists: two zones survive one of their two possible zone failures.
- Limiting blast radius — cells, shuffle sharding — is often cheaper than raising component reliability.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Enumerate the failure causes you intend to survive, from process crash upward.
- • For each, determine the set of components that share that cause.
- • Place redundant copies so that a majority survives the loss of any one domain at the level you chose.
- • Trace start-up dependencies as well as steady-state ones, because recovery has more dependencies than operation.
- • Partition the user population so that one failure reaches a bounded fraction of it.
- • A shared configuration or secrets service takes down every zone simultaneously.
- • A control plane failure prevents replacement instances from starting, so a small failure cannot be repaired.
- • A deploy propagates a fault to every domain within minutes.
- • A quorum placed across two zones loses its majority when the wrong zone fails.
- • A dependency needed only at start-up is unavailable exactly when everything is restarting.
- • Correlated zone failure through a shared service: instances in three zones fail together. The operator sees a fleet-wide error rate with no zone-level infrastructure alert, and spends the first twenty minutes looking at the wrong layer.
- • Unrecoverable small failure: a handful of instances die and cannot be replaced because the control plane or registry is down. The operator sees capacity draining steadily with no new instances arriving.
- • Deploy-propagated outage: a change reaches all zones in minutes. The operator sees error rate tracking rollout percentage, which is the fastest diagnostic available and requires deploy markers to be visible.
- • Wrong-zone quorum loss: a two-zone cluster loses its majority side and goes read-only. The operator sees writes failing while every node reports healthy, because the surviving node is healthy and alone.
- • Single-tenant blast: one tenant’s traffic pattern saturates a shared pool and every tenant suffers. The operator sees global latency degradation traceable to one customer id.
- • Quorum placement is the coordination consequence of fault domains: a quorum must be reachable within a surviving domain, which is why three domains rather than two is a structural requirement rather than a preference.
- • Cross-region quorums make every write pay the inter-region round trip, so surviving a region failure with strong consistency has a latency floor set by geography.
- • Cell-based designs avoid cross-cell coordination entirely, which is what makes them robust — and also what makes cross-cell operations awkward.
- • Components in unaffected domains continue normally, provided they have no dependency in the affected one.
- • A quorum that retains a majority continues to accept writes; one that does not becomes read-only or unavailable.
- • Recovery capacity depends on the control plane, which is frequently in a domain nobody checked.
- • Detect: alert on domain-level failure explicitly, not only on aggregate error rates, so the level of the problem is immediately clear.
- • Contain: fail traffic away from the affected domain, and ensure the failover path does not depend on it.
- • Recover: restore redundancy first — running degraded with no spare domain is the state in which a second failure is catastrophic.
- • Reconcile: after a domain returns, resynchronise before returning it to service, rather than routing to a stale replica.
- • Verify: exercise domain failure deliberately, because a failover path that has never run is a hypothesis.
- • Instance and replica distribution across domains as a monitored fact, not a deployment-time intention — drift here is silent and common.
- • Quorum survivability: for each domain, whether a majority remains if it is lost right now.
- • Dependency graph including start-up-only dependencies, which is where recovery failures hide.
- • Blast radius per failure class: what fraction of users a single cell, shard or instance failure affects.
- • Any time redundancy is being added, because redundancy within a domain buys nothing against that domain’s failures.
- • During capacity planning, since surviving a domain loss requires headroom in the remaining domains.
- • Spreading a latency-sensitive, chatty system across zones adds a round trip to every internal call for an availability gain that may be smaller than the latency cost.
- • Cell-based isolation for a small system multiplies infrastructure and operational surface before there is enough traffic to justify it.
- • Accept the domain failure and plan the recovery instead of the redundancy — for many systems, an hour of downtime once a year is cheaper than multi-zone complexity.
- • Use a managed service that has already solved domain placement, and consume its stated availability.
- • Reduce blast radius rather than increasing reliability: shuffle sharding costs a routing rule and protects against causes you did not predict.
- • Make the system stateless so recovery is re-creation rather than repair, which turns a domain failure into a capacity problem.
What fails together
| Survives one machine | Survives one zone | Survives the control plane | |
|---|---|---|---|
| 5 replicas across 2 zonesassumption | yes — quorum 3 of 5 holds | only if the small zone fails | not modelled here |
What people believe, and what is true
We are multi-zone, so we survive a zone failure.
Only if no shared dependency spans the zones and a quorum survives. Both are commonly false, and neither is visible on the deployment diagram.
Two zones is enough for a quorum.
A three-node quorum across two zones survives one of the two possible zone failures. Three zones survives either.
More replicas means more fault tolerance.
Only across domains. Six replicas in one zone tolerate five machine failures and zero zone failures.
The control plane is not part of my system.
It is the fault domain that decides whether you can recover. A system that survives the fault and cannot replace capacity has a long outage, not a short one.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A fault domain is what fails together. Place redundancy across domains, and remember the invisible ones: shared config, control planes, and the deploy pipeline.
Practical
Write down, per component, its start-up dependencies and its steady-state dependencies. Check that a majority of every quorum survives the loss of any one zone — which needs three zones, not two. Then measure how much of your user base a single failure can reach, and reduce it with cells or shuffle sharding before buying another nine.
Advanced
The subtle version is metastable failure: a system that is stable under normal load and stable under overload, with a barrier between them that a fault domain failure can push it over. Losing a zone redistributes its load onto the remaining zones, which raises their latency, which raises retry rates, which raises load further. The system does not recover when the zone returns, because the retry load is now self-sustaining. This is why capacity planning for domain loss must account for the retry amplification the loss creates, not merely for the redistributed steady-state traffic — and why load shedding, not extra capacity, is often the thing that breaks the cycle.
Apply it
- ⚡ Your system is deployed across three zones and goes down entirely. Walk through the hypotheses in the order you would test them.
- ⚡ Design the placement for a five-node consensus cluster that must survive one zone failure and tolerate one additional machine failure.
- 💬 Why is a three-node quorum across two availability zones not zone-fault-tolerant?
- 💬 Name three fault domains that do not appear on a typical architecture diagram.
- 💬 What is the difference between a start-up dependency and a steady-state dependency, and why does it matter during an incident?