Reliability & Disaster Recovery
Redundancy that is actually redundant: failure domains from process to region, multi-zone and multi-region topologies, active-passive against active-active and the data problem it creates, RPO and RTO as requirements rather than adjectives, backup strategy, and the restore you have never tested.
Reliability is four questions asked of every component: when this fails, is there a redundant one, is traffic rerouted, is state preserved, and how do we get back? A "yes" that has never been demonstrated is a "no".
Q · When a piece of this infrastructure dies, what happens next — and who or what makes it happen?
The standard shape: one region, an application spread across two zones, a load balancer in front. What it protects against, what it does not, and why database redundancy is a separate problem with a separate answer.
Q · What is the smallest topology that survives losing a host or a zone without a customer noticing?
The ladder from process to region, and the one question that finds most reliability bugs: are all the replicas actually in the same failure domain? Three replicas on three VMs in one zone is one failure domain wearing a disguise.
Q · When something fails, what else fails with it — and are my replicas genuinely independent or only visually separate?
Spreading a workload across the independent facilities inside one region. The reliability is real, the latency is small but not zero, the data placement is subtle, and the cross-zone transfer meter is the line item nobody predicted.
Q · What does spreading a workload across zones inside one region actually buy, and what does it charge for?
Running in more than one geographic region. It is the answer to latency, to regional failure and to data-residency law — and "multi-region" on its own says nothing about availability. A multi-region deployment with one primary database in one region is single-region availability with a bigger bill.
Q · What does a second region actually buy, and which of those things does it fail to buy unless you also solve the data problem?
One region serves; another waits. Simple, cheap compared with active-active, and built entirely around a failover path that almost nobody tests — which is why the standby so often fails on the one day it is needed.
Q · If one region serves everything and another waits, what exactly happens at the moment the first one fails?
Both regions serve traffic at the same time. It removes failover from the critical path and replaces it with a permanent distributed-data problem: conflicts, ordering, routing and consistency, all live, all the time.
Q · What does it take for two regions to serve traffic simultaneously, and what problem do you take on permanently in exchange?
What you do when redundancy was not enough: backup, restore, failover, and the two numbers that decide which of them you need. DR is the plan for the failures that take the whole environment, including the ones you caused yourself.
Q · When the environment itself is gone — a region, an account, a deletion nobody meant to run — what is the documented path back, and how long does it take?
Two numbers that turn "we need it to be reliable" into an architecture. RPO is how much data you can afford to lose; RTO is how long you can afford to be down. Derive the design from the numbers, never the numbers from the design.
Q · How much data may we lose, and how long may we be down — and what does each answer force us to build?
A backup is only a backup if it is in a separate failure domain. Same account, same region, deletable by the same credentials that just got compromised — that is a copy, and it does not survive the most common causes of data loss.
Q · Where does the second copy live, who can delete it, and how far back can it take us?
A backup you have never restored is not proven recovery. It is a file with a hopeful name. The only way to know whether you can recover is to recover, on a schedule, with a stopwatch.
Q · How do you know your backups work — and what is the only answer that is not a guess?