The question this answers
When a piece of this infrastructure dies, what happens next — and who or what makes it happen?
The service is expected to keep serving through the ordinary bad luck of running in a data center: a host dies, a disk fills, a certificate expires, a zone loses power. None of these are unusual, and none of them should be a customer-visible outage.
A component-by-component answer to "and then what?", so that failures produce degraded capacity and a page, rather than an outage and a discovery.
The four questions
Reliability work is not a feeling about a diagram. It is an inventory: take every component, assume it is gone, and answer four questions in order. Is there a redundant component that can take the load? Is traffic actually rerouted to it, by what mechanism, in how long? Is the state preserved, or did it live only on the thing that died? And what is the path back to normal, including who runs it?
Most infrastructure passes question one and fails question two. Two application instances exist, but both are registered against a load balancer whose health check has an 90-second detection window — so "redundant" means 90 seconds of failed requests. Or the standby database exists and promoting it is a manual runbook nobody has run this year. Redundancy without automatic, tested rerouting is a component you pay for and cannot use in the moment you need it.
Question three is where the honest answers get uncomfortable. Stateless tiers pass trivially. Anything holding state — a database, a queue, a session store, a cache warmed over hours, a file written to local disk — needs a real answer about what survives. And question four is the one that is almost never written down: the recovery path, its owner, and the last time anyone executed it.
Redundancy that is not redundant
The most common finding in a reliability review is a component that appears twice on the diagram and once in reality. Two application instances placed by a scheduler that put them on the same host. Two database nodes in the same zone. Two load balancer targets that both resolve their configuration from one config service. Three replicas whose shared dependency is a single NAT device in one zone — see Failure Domains for the systematic version of this check.
A second class is redundancy that exists but cannot be reached in time. A warm standby that requires a DNS change with a 300-second TTL cached by clients that ignore TTLs. A replica that must be promoted by a human who is asleep. A backup that lives in the same account that was just compromised.
The topology below is the shape of a design that survives the review. Every stateful component names its redundancy and its rerouting mechanism, and the components that are deliberately single are marked as accepted risk rather than overlooked. "We accept that a region failure takes us down for four hours" is a legitimate engineering position; "we never thought about it" is not.
Reliability is bought, and the price is not linear
Every increment of reliability has a price, and the prices are wildly uneven. Going from one instance to two roughly doubles compute cost and removes the single largest cause of outages. Going from two zones to three adds perhaps 50% and removes a much rarer failure. Going from one region to two can more than double total cost, add a data-consistency problem you did not have, and — done badly — reduce reliability by adding a failure mode nobody understands.
So the discipline is to spend where the failure probability actually is. Host failures are common and cheap to survive. Zone failures are rare and moderately priced. Region failures are very rare and extremely expensive to survive properly. Certificate expiry, quota exhaustion, a bad deploy and a mistaken terraform apply are far more likely than any of them, and are addressed with process and automation rather than with capacity.
That last point deserves emphasis, because it is where reliability budgets are most often misallocated. Most outages are not hardware. They are changes: a deploy, a configuration edit, a schema migration, an expired credential. A team that has built multi-region redundancy but deploys without canaries has bought insurance against the rare failure while leaving the common one uncovered — see Canary: Let 5% of Traffic Find the Bug and Drift: When the File and Reality Disagree.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- For every component ask: redundant? rerouted automatically? state preserved? recovery path tested? A gap at any step is where the outage will be.
- Redundancy without automatic rerouting is capacity you pay for and cannot use during the incident.
- A component that appears twice on the diagram may be single in reality — same host, same zone, same config dependency.
- Most outages are changes, not hardware. Deploy safety often buys more reliability per euro than another zone.
- Reliability increments are priced very unevenly; spend where the failure probability actually is.
- An untested recovery path is not a recovery path, it is a document.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Failure detection: health checks, heartbeats, control-plane monitoring — each with its own detection window.
- • Rerouting: the load balancer removes a target, DNS is updated, or a replica is promoted and clients reconnect.
- • State handling: replication keeps a second copy current; without it, state that lived on the failed component is gone.
- • Capacity absorption: survivors must have headroom to take the failed component's load, or failover simply moves the outage.
- • Recovery: the failed component is replaced (immutable) or repaired (mutable), and the system returns to its intended shape.
- • Keep a written failure inventory — component, failure mode, detection, response, recovery owner. Review it when the architecture changes, not annually.
- • Verify the N-1 capacity assumption: if losing a zone means the survivors take 100% more traffic, they must actually have that headroom, and autoscaling will not deliver it in time (Startup Time & Cold Start).
- • Own the detection windows. Total downtime is detection + decision + failover + recovery, and detection is usually the largest term.
- • Test failure deliberately: kill an instance, a zone, a dependency, during business hours, with people watching.
- • Record accepted risks explicitly. "Region failure means four hours down" written on the design is a decision; unwritten, it is a surprise.
- • Correlated failure: both "redundant" components share a host, a zone, a config service or a credential.
- • Failover that never triggers because detection was configured for a fast crash and the failure was a slow degradation.
- • Failover that triggers into insufficient capacity, so the survivors fall over in turn — the classic cascading outage.
- • State loss on components nobody classified as stateful: local disk caches, in-memory sessions, a queue with no persistence.
- • Recovery blocked by the failure itself — the runbook lives in a wiki hosted in the region that is down.
- • Retry storms during recovery: every client retries at once and the newly restored component is immediately overwhelmed.
- • More components means more failures per unit time. At a hundred instances, a host failure is a weekly event and must be routine rather than an incident.
- • Failure detection and recovery machinery must scale too: a control plane that reconciles a thousand objects behaves differently than one reconciling ten.
- • The N-1 headroom cost falls as the fleet grows — losing one of three zones costs 50% headroom, losing one of twenty instances costs 5%.
- • Blast radius grows with shared infrastructure. The larger the shared control plane, config service or identity provider, the more a single failure takes with it.
- • Availability is a security property. A denial-of-service condition and an outage are the same event from the user's side (Public Exposure, Read With Context).
- • Failover paths often skip controls: an emergency read-replica promotion that grants broad credentials to a human is a real, recurring pattern.
- • A compromised credential can be a reliability event — anything with delete permission on infrastructure is a single point of failure regardless of how many replicas exist (Least Privilege in Infrastructure).
- • Redundant copies multiply the places sensitive data lives. Each standby, each backup, each region needs the same encryption and access controls as the primary.
- • Redundancy is mostly a fixed cost: a standby is billed whether or not it is ever used.
- • Cross-zone and cross-region data transfer are the meters that grow with redundancy and are consistently underestimated.
- • N-1 headroom is idle capacity by design — it is the difference between right-sizing and under-sizing (Right-Sizing Without Causing an Outage).
- • Testing costs engineering time and prevents the expensive class of failure, which makes it the best-value line on the reliability budget.
- • Redundancy status as a monitored fact: replica count, standby lag, zone distribution — alert when the *design* is violated, not only when serving fails.
- • Failover events and their durations, with each one treated as data about the mechanism rather than only as an incident.
- • Capacity headroom per failure domain, so you know before the incident whether N-1 is real.
- • Time-to-detect per failure class, which is the term you can most often shorten cheaply.
- • The signal that lies: an aggregate availability percentage. It hides that one component has been the cause every time.
- • Accept the downtime. For an internal tool used by forty people in one time zone, a four-hour recovery is a rational choice and every redundancy euro is better spent elsewhere. Write it down and move on.
- • A managed service that includes redundancy in its price — you are buying someone else's tested failover rather than building an untested one (Managed Databases).
- • Backup and restore as the whole strategy, with a documented and rehearsed RTO of hours (Disaster Recovery). Far simpler than standby infrastructure and honest about its limits.
- • Deploy safety first: canaries, rollback, change review. For most teams this removes more downtime per euro than any amount of redundant capacity.
- • Every redundant component doubles a cost and adds a failover mechanism that can itself misbehave.
- • Automatic failover removes human latency and introduces the risk of failing over for the wrong reason, at the wrong time.
- • More failure domains means more reliability and more operational surface: more configuration, more drift, more places to be inconsistent.
- • The most reliable systems are often the simplest ones. Complexity added in the name of reliability frequently reduces it (Scoring Operational Complexity).
What people believe, and what is true
We have two of everything, so we are highly available.
Two of everything in one failure domain is one of everything. And redundancy without automatic, tested rerouting does not help during the incident.
Reliability is an infrastructure problem.
Most outages are changes. Deploy practice, rollback and change review are reliability engineering as much as redundant capacity.
Adding components makes the system more reliable.
Each added component is another thing that fails, and complexity itself is a leading cause of outages.