Active-Passive: Simple to Reason About, Rarely Tested

The word hides a spectrum, and the differences dominate your recovery time far more than any architectural subtlety.

A cold standby is data only: backups in the second region, no compute running. Recovery means provisioning everything, which is hours, and it is the honest choice when the business genuinely tolerates hours. A warm standby runs the databases and replicates, but the application tier is scaled to zero or near it. A hot standby runs everything at production scale and receives no user traffic. A hot standby serving reads runs everything and takes read traffic, which is the only variant continuously proving that it works — and is therefore the only one whose failover has a good chance of succeeding.

That last observation is the most useful thing in this lesson. The reliability of a standby is roughly proportional to how much of it is used in normal operation. Everything else is a plan, and plans decay.

The runbook says "fail over to the standby". What happens is a chain, and each link has its own latency, its own failure probability and its own owner — so the total time is the sum, not the maximum, and the total success probability is the product, not the best link.

Write these out with real numbers for your own system and the RTO in the disaster recovery document usually turns out to be a third of the truth. The detection step alone is typically the largest term, because you cannot set it aggressively: a detector fast enough to catch a real outage in thirty seconds is also fast enough to fire on a routine network blip, and a spurious failover is worse than a slow one.

Under asynchronous replication, the writes sitting in the replication lag window at the moment of failure are gone. Not delayed — gone, unless the dead region comes back and you reconcile manually. And those writes were acknowledged to users. Someone was told their order was placed.

So state the RPO as a measured quantity: "replication lag p99 is 3.2 seconds, therefore our worst-case data loss is roughly 3.2 seconds of writes, which at peak is about 900 orders." That sentence is worth more than any diagram, because it is the sentence that gets someone to fund synchronous replication or to accept the risk explicitly.

The lag is not constant. It climbs under exactly the conditions that precede failures: a load spike, a bulk import, a slow disk in the standby, a network degradation. The correlation is unhelpfully in the wrong direction — your data loss window is widest at the moment you are most likely to need it. Alert on the lag itself, not on the health of the write path, because the write path will be perfectly healthy the whole time.

Synchronous replication removes the loss and buys it back as latency and availability: every write now waits for the far region, which is the RTT from [[speed-of-light]], and a standby that is merely slow now stalls the primary. [[synchronous-replication]] covers the mechanics; the geo-specific consequence is that at 90–200 ms per write, synchronous cross-region replication is affordable for a ledger and not for a session store, so most systems end up doing both for different data.

Promotion is not an administrative act. It is a claim that there is now exactly one primary, and that claim can be false in the one way that matters: the old primary is alive, unreachable from you, and still accepting writes from users who can reach it.

This is why fencing must precede promotion, not follow it. The safe sequence is: revoke the old primary’s ability to commit — by lease expiry, by storage-level token rejection, by taking away the network path or the credential — *and then* promote. If you promote first and fence later, you have created two primaries for the length of the gap, and every write in that gap is a divergence that no automatic mechanism will merge. [[fencing-tokens]] is the general treatment and [[split-brain]] is the failure.

A human in the loop is the usual compromise and a defensible one. It costs minutes of RTO and buys a judgement that no health check can make. What is not defensible is an automatic failover with no fencing, which converts a partition — recoverable — into a divergence — not recoverable. If your automation cannot fence, it should page instead of promoting.

The dead region comes back. It has a database that diverged from the promoted one for however long it kept accepting writes, plus whatever it had that never replicated. Now what?

The honest answer for most teams is that there is no procedure, so the recovered region is rebuilt from the new primary and the divergent writes are lost — a second data-loss event, occurring after the incident is declared resolved, and usually not counted in it. Doing better means capturing the old primary’s unreplicated tail before wiping it, and reconciling it as a business exercise, which is [[reconciliation]] and is genuinely slow work.

The other trap is failing back too eagerly. The original region is "the main one" for organisational reasons, so traffic is moved back during business hours, which is a second unrehearsed failover, in the opposite direction, on a region whose state you are not sure of. Failing back is not the reverse of failing over. It deserves its own plan and its own maintenance window, and often the right answer is to stay where you are and rename the regions in the documentation.

The failure mode of active-passive is not conceptual, it is entropic. The standby drifts: a config change applied to production only, a certificate that expired, an IAM role never created there, a capacity quota that was never raised, a schema migration applied to the primary and not the replica, a secret rotated in one place, a dependency that only whitelisted the primary region’s egress addresses. None of these produce a signal until the failover.

There are exactly two effective countermeasures, and neither is a document. Route real traffic through it — even 5% of reads keeps the config, capacity, certificates and dependency allowlists continuously honest. Fail over on purpose, on a schedule — quarterly, in business hours, announced, with the ability to abort. A failover that has been performed six weeks ago is a routine operation; one that has never been performed is an experiment being run for the first time during an outage. [[chaos-engineering]] and [[fault-injection]] are the disciplines; Cloud’s restore-testing is the same argument applied to backups.