The question this answers
When something fails, what else fails with it — and are my replicas genuinely independent or only visually separate?
The service claims to tolerate any single failure. That claim is only true if the redundant copies do not share whatever is about to break. Somebody has to check, and the check is not "count the boxes on the diagram".
A vocabulary and a procedure for asking what fails together, so redundancy is verified against a named boundary instead of assumed from a picture.
The ladder: process, host, rack, zone, region
A failure domain is the set of things that fail together. They nest: a process failure takes one process, a host failure takes every process on it, a rack failure takes every host in it, a zone failure takes every rack in the facility, and a region failure takes every zone. Each rung is less likely than the one below and takes more with it.
Redundancy only helps if the copies sit in *different* domains at the level you are protecting against. Two processes on one host survive a process crash and not a host failure. Two hosts in one rack survive a host failure and not a power distribution unit. Two zones survive a facility event and not a regional control-plane outage. The question is never "do we have redundancy" but "redundancy against which rung".
Cloud providers expose this ladder deliberately, and imperfectly. Zones are the boundary they commit to: independent power, cooling and networking within a region. Racks are usually invisible, approximated by placement or spread policies. The process and host rungs are yours to manage through scheduling. Regions are the outermost boundary and the only one that survives a provider-wide control-plane failure in one region — see Regions and Availability Zones.
Ask it out loud: are all the replicas actually in the same failure domain?
This is the question that finds the bug, and it must be asked as a question rather than assumed as an answer. Three replicas is a number that feels safe. Three replicas on three VMs in one zone is one failure domain wearing a disguise: it survives three separate host failures and none of the events that take a facility.
The disguises are consistent enough to check for. Replicas scheduled without an anti-affinity rule land on whichever nodes had room, which is frequently the same node. A "multi-zone" cluster whose node pool is defined in one zone. A stateful set whose volumes are all provisioned in the zone the first pod happened to start in — storage is often zonal, which quietly pins the pod. A queue consumer group scaled to six with all six in the same subnet.
Non-physical shared fate is harder and just as real. Every replica reads the same config service at startup, so that service is a failure domain containing all of them. Every replica uses one certificate that expires simultaneously. Every replica runs the same image, so a bad image is a fleet-wide event that no amount of geographic spread addresses. This is why deploy safety belongs in a reliability conversation: your deployment pipeline is a failure domain that spans every zone and region you own.
The domains that are not physical
Physical domains are the ones with names and dashboards. The ones that cause the memorable outages usually have neither. A shared credential is a failure domain: when it expires or is revoked, everything using it fails together, everywhere, instantly. A shared dependency — one identity provider, one DNS zone, one package registry — is a failure domain spanning every region you carefully separated.
Change is the largest of them. One deployment pipeline pushes one image to every zone and region; a bad image is a global failure with perfect correlation. This is precisely why staged rollouts exist: Canary: Let 5% of Traffic Find the Bug and Rolling Deployment and the Compatibility It Demands deliberately break the correlation by making the change arrive at different times in different places. A team with three regions and an all-at-once deploy has geographic redundancy and no protection against its most likely failure.
The checklist below is what a review actually consists of. It takes an hour and it is the highest-value reliability exercise available to most teams.
| Shared thing | How to check | What it means if shared | Typical fix |
|---|---|---|---|
| Host / node | Inspect actual placement, not the desired count | One host failure takes several replicas | Anti-affinity or spread placement rules |
| Zone | Group replicas by zone and count the groups | A facility event takes the whole service | Node pools or scaling groups per zone |
| Storage volume locality | Check whether volumes are zonal or regional | Replicas are pinned and cannot reschedule across zones | Regional/replicated storage, or shared object storage |
| Config or secret source | List what every replica reads at startup | That source is a failure domain containing the fleet | Cache locally, fail open on read errors, replicate the source |
| Certificate / credential | Find the expiry and the issuing path | Simultaneous expiry is a global outage on a known date | Automated rotation plus expiry alerting well ahead |
| Deployment pipeline | Ask how a bad image reaches production | A change is a perfectly correlated global failure | Staged rollout, canary, automated rollback |
| Cloud account / identity | Ask what one compromised admin credential reaches | Every region shares one blast radius | Separate accounts per environment, restricted destructive permissions |
Key points
- A failure domain is the set of things that fail together; process, host, rack, zone and region nest inside one another.
- Ask explicitly: are all the replicas actually in the same failure domain? Three replicas in one zone is one failure domain.
- Zonal storage silently pins a workload to a zone even when the scheduler could otherwise move it.
- Non-physical domains — shared config, shared credentials, one deployment pipeline — are frequently larger than any physical one.
- Your deployment pipeline spans every region you own, which is why staged rollouts are reliability engineering.
- The cloud account itself is a failure domain, and usually the biggest one.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Providers expose zone identity as metadata; schedulers and scaling groups use it as a placement key.
- • Anti-affinity and spread policies instruct the scheduler to place replicas in different domains rather than wherever capacity is convenient.
- • Scaling groups spanning multiple zones rebalance placement automatically as instances are added and removed.
- • Zonal storage is attached within one zone; regional or replicated storage is what allows a workload to move across zones.
- • Quorum systems need their members placed across an odd number of domains so a single domain loss cannot break the majority (Stateful Workloads: Databases Are Not Stateless APIs).
- • Audit real placement, not intent.
desired: 3tells you nothing about where they landed. - • Alert when the distribution violates the design — for example, when more than one third of replicas land in one zone.
- • Maintain an inventory of shared dependencies and treat each as a named failure domain with an owner.
- • Track certificate and credential expiry centrally. Simultaneous expiry is the most predictable outage there is, and it still happens constantly.
- • Include the deployment pipeline in failure-domain reviews and require staged rollout for anything with a wide blast radius.
- • Silent co-location: the scheduler placed everything on one node because that is where capacity was, and nothing complained.
- • Zonal volume pinning: replicas cannot be rescheduled after a zone failure because their storage is gone with it.
- • Quorum loss: three of five members were in one zone, so losing it destroys the majority and the cluster goes read-only.
- • Correlated expiry: one certificate across every service, expiring at the same second in every region.
- • A bad image deployed everywhere at once — perfect geographic redundancy, zero protection.
- • A shared control-plane dependency in one zone, so zone loss prevents replacing the capacity you just lost.
- • More replicas do not create more independence; distribution across domains does. Ten replicas in one zone are still one zone.
- • Spreading across three zones rather than two reduces the headroom cost of losing one from 50% to 33%.
- • Cross-domain spread costs latency and data transfer, and both grow with how chatty the tiers are.
- • Quorum systems have hard constraints: majorities must survive the loss of one domain, which sets a minimum number of domains rather than a minimum number of nodes.
- • Blast radius is the security version of the same concept: what one compromised identity reaches is exactly a failure domain (Least Privilege in Infrastructure).
- • Separate accounts or projects per environment create a domain boundary that survives both a mistake and a compromise — production and staging sharing one account is a shared fate.
- • A shared secret store is a failure domain and a security boundary at once; its availability is now part of your availability (Secrets in Infrastructure).
- • Isolation for reliability and isolation for security usually want the same boundaries, which makes the argument for either one cheaper to win.
- • Spreading across zones is mostly free in compute and costs in cross-zone data transfer.
- • Anti-affinity can force less efficient packing: reserving room on separate nodes means paying for capacity you cannot fill.
- • Regional or replicated storage costs more than zonal storage, and buys the ability to reschedule across zones.
- • Separate accounts per environment add administrative overhead and occasionally lose you volume discounts, in exchange for a hard boundary.
- • Replica distribution per zone and per node, as a monitored fact with an alert when it drifts from the design.
- • Placement policy violations reported by the scheduler — these are usually available and rarely watched.
- • Certificate and credential expiry dates, with alerts weeks ahead rather than hours.
- • Which components share a dependency, kept current in a document or a service map. This is the artifact that lets a new engineer answer the question.
- • The signal that lies: replica count. It is the number people check and it says nothing about independence.
- • Accept a single failure domain and cover it with a fast, tested restore. For many workloads a documented 30-minute recovery is cheaper and more honest than spreading a system that was never designed to be spread.
- • Use a managed service that handles domain placement internally — a regional managed database or object store is someone else's multi-domain problem, already tested.
- • Spread only the tier that matters. A stateless API across three zones with a single-zone batch job is a coherent design if the batch job may be down for an hour.
- • For a three-person team, one zone plus good backups beats a multi-zone design nobody has capacity to operate correctly.
- • Wider spread means better independence, more cross-domain traffic, higher latency and more cost.
- • Strict anti-affinity guarantees separation and can leave pods unschedulable when capacity is tight — a reliability rule that causes an availability incident.
- • Separate accounts contain blast radius and multiply the operational surface: more configuration, more drift, more places to be inconsistent.
- • Removing shared dependencies improves independence and usually means duplicating something, which costs money and creates consistency problems.
Count the failure domains, not the replicas
Failing over also assumes the surviving domains have room for the traffic the dead one was carrying. Spread 3 replicas over 1 domains and each survivor must absorb its share, so sizing every replica to its own average load is how one zone failure takes down the ones that were still healthy.
What people believe, and what is true
Three replicas means we tolerate failures.
It means we tolerate failures of whatever domain the replicas are separated by. If they share a node or a zone, that is the only thing you tolerate.
Failure domains are physical.
Shared credentials, shared config services and one deployment pipeline are failure domains too, and they are usually wider than any physical one.
The scheduler will spread things sensibly.
A scheduler optimizes for the constraints it was given. With no spread constraint, packing everything onto one node is a correct answer to the question it was asked.