Region Failover
Five questions decide whether a failover works: is the data there, can traffic move, is there capacity, are config and secrets present, are dependencies reachable.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What has to be true in the target region before shifting traffic to it can possibly work?
Failover is usually rehearsed as a routing change. Routing is the easy part and the last part; everything that makes it succeed had to be true before the event.
We have a standby region. If the primary goes down we update DNS to point at the secondary and traffic follows.
The standby region has data that is behind the primary, or has no writable database at all because the replica was never promoted.
- The standby region has data that is behind the primary, or has no writable database at all because the replica was never promoted.
- Capacity in the target was sized for its share of normal traffic, so it saturates as soon as it receives all of it (Capacity During Failover).
- Configuration and secrets exist only in the primary region, so services start and immediately fail to authenticate to anything.
- The application in the target still calls dependencies in the failed region — a queue, an object store, an internal service, a third-party endpoint pinned to a regional address.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Failover moves the *serving role*, and the serving role has five prerequisites. Traffic routing is only the trigger; the other four determine whether the target can actually serve.
- Data availability means a copy in the target that is current enough for the recovery point *and* writable — which for asynchronous replicas means a promotion step, and promotion is not reversible without care.
- Traffic movement can be a DNS change, an anycast or global load balancer change, or a client-side configuration change. They differ enormously in how fast they take effect and how much of the tail you control (DNS in Production).
- Capacity is arithmetic: if two regions normally each serve half the traffic, either one must be able to serve all of it, which means each runs at half its capacity in normal times or must scale up during the event — and scaling up takes time you are already spending.
- Dependencies are the most commonly missed prerequisite: a regionally-pinned endpoint in the failed region turns a successful failover into an outage with better dashboards.
Five prerequisites, checked before the traffic moves
Each of these fails independently, and each has a pre-check you can run on an ordinary day. A failover runbook that starts with the routing change has skipped the four that decide the outcome.
- 1Data
Confirm a copy exists in the target, is current enough for the recovery point, and can be made writable.
fails by Promoting a lagging replica and silently losing acknowledged writes.
evidence Live replication lag for the target, and a rehearsed promotion with the resulting data gap observed.
- 2Routing
Move traffic by DNS, global load balancer or client configuration.
fails by A long tail of clients still resolving or holding connections to the old region.
evidence Request share by region on a dashboard, watched until the tail is acceptably small.
- 3Capacity
Ensure the target can serve the whole load, now or after a scaling step.
fails by Saturation shortly after the shift; the outage continues with a new cause.
evidence An observed full-load period in the target, or a scaling path with a measured lead time (Headroom).
- 4Config and secrets
Ensure configuration, credentials and identity are present and valid in the target.
fails by Services start, then fail authentication against every dependency.
evidence A pre-check that the target can read each required secret and pass a startup validation (Validate at Startup, Fail Clearly).
- 5Dependencies
Ensure everything the application calls is reachable from the target.
fails by A regionally-pinned queue, bucket, internal service or third-party endpoint still points at the failed region.
evidence A dependency inventory per service, with the resolved endpoint recorded from inside the target region.
Run the pre-check on a schedule, not during the incident. Its value is that it fails on a Tuesday.
What moves and what does not
Drawing the failover makes the asymmetry visible: the routing layer is global and changes quickly, the data layer is regional and changes slowly, and the dependency edges are where people are surprised.
- The replication edge carries the recovery point: whatever has not crossed it when you promote is the data you lose.
- The two dashed-looking edges from the target app tier are the ones that turn a clean failover into a partial outage.
- The routing edge has a tail. Plan for both regions receiving traffic simultaneously for a period, which means the application must tolerate it (Version Coexistence: N and N+1, in Both Directions).
How failovers actually fail
Notice how many rows describe damage done by the failover rather than by the original event. This is why the procedure is rehearsed and why the decision stays human where the signal is ambiguous.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Promote an asynchronous replica during a primary outage | Recent orders missing after recovery | Unreplicated writes were acknowledged to users and did not survive promotion | Show replication lag at decision time; capture what was lost for reconciliation; consider synchronous replication if the loss is unacceptable |
| Primary is partitioned but alive | Conflicting writes in both regions | Split brain — no fencing of the old primary | Fence before promoting: revoke the old primary's ability to accept writes as a required step, not an afterthought |
| Traffic shifted to a target sized for half the load | Latency and errors climb within minutes of the shift | Capacity planning assumed steady-state split, not failure state | Size for full load or pre-scale as a required pre-check step (Capacity During Failover) |
| Application starts in target region | Immediate authentication failures to every dependency | Secrets were never replicated, or were rotated only in the primary | Replicate secrets as part of rotation and validate at startup (Rotation That Applications Survive) |
| Primary recovers; team fails back immediately | Writes made during the event disappear | Replication was still flowing the original direction, or was re-established backwards | Treat failback as a planned change with its own pre-checks and reconciliation step |
How to do it properly
Most important first.
- Answer the five questions explicitly per system and keep the answers in the runbook: data, routing, capacity, config and secrets, dependencies.
- Decide in advance whether failover is manual or automatic, and if automatic, on what unambiguous signal. Automatic failover on a flapping signal produces its own outage (The Automation Trap).
- Shorten the routing tail deliberately — appropriate DNS TTLs, health-checked global load balancing, and a plan for clients that keep connections open (Draining: Stopping Without Dropping).
- Rehearse by actually shifting production traffic on an ordinary day, at a small percentage first. A failover that has only been simulated is an untested procedure.
- Write the failback plan at the same time as the failover plan, including how divergent writes are reconciled.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rehearsed procedure with pre-checks and a verified target contains it. An unrehearsed one converts a regional event into a global one, because a botched failover usually breaks the healthy side too.
What can go wrong
- Promotion of a lagging replica loses writes that had been acknowledged to users, and the loss is discovered later as customer complaints rather than as an alert.
- Split brain: the primary is not actually dead, and two regions accept writes to the same logical dataset.
- Automated failover triggers on a monitoring failure rather than a service failure.
- Failback is attempted casually and loses the writes made while running in the recovery region.
- "We changed DNS, so traffic has moved." Some of it has. Caching resolvers, client-side caches and long-lived connections keep sending to the old target for a tail you do not control (DNS in Production).
- "The standby is up, so we are ready." Up is not the same as current, writable, capacity-ready, configured and connected.
- "Failover is the recovery." Failover buys availability. The event is not over until you have failed back or accepted the new region as primary, deliberately.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A dated record of a real traffic shift, not a tabletop exercise, with per-phase timings.
- A capacity assertion for the target: it has been observed serving the full production load, or has a scaling path with a measured lead time.
- The rollback of a failover is a failback, and it is the harder direction: the recovery region has accepted writes the original has not.
- Re-establish replication in the new direction, let it catch up, then move the serving role back deliberately in a low-traffic window — never as an urgent reflex once the primary looks healthy.
- If writes diverged, reconciliation is application work, not a database operation. Plan it before you need it (Partial and Logical Data Recovery).
- Automate: health checking, traffic shifting mechanics, replica promotion steps, scaling the target, and the verification suite.
- Automate the pre-checks — data currency, capacity, config presence, dependency reachability — so the human decision is informed in seconds rather than assembled during the incident.
- Keep human: the decision to fail over, unless the signal is genuinely unambiguous and the cost of a false positive is low; and always the decision to fail back.
- Keeping a target able to absorb full traffic means paying for capacity that is idle in normal operation (Idle Capacity).
- Automatic failover shortens recovery time and introduces a new class of incident caused by the automation itself.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- CLOUD-SPECIFICProviders differ in whether a managed database offers cross-region replicas that can be promoted, how long promotion takes, and whether global load balancing is anycast or DNS-based. On some services a "cross-region" option is only a snapshot copy, which makes failover a restore and changes the achievable recovery time completely.
- DATABASE-SPECIFICPromotion semantics differ sharply: an asynchronous replica may be promoted with unreplicated writes lost, a synchronous standby avoids that loss at a write-latency cost, and a quorum-based system may fail over automatically but only where a majority survives. Know which one you have before the event, not during it.
- SCALE-SPECIFICBelow a certain size, an honest answer is that you do not fail over: you restore into another region and accept the recovery time. That is a legitimate, cheaper choice, and much better than an unrehearsed failover.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — fencing, quorum and split brain: why "is the primary really dead" is not answerable from inside the system, and what that forces on the procedure.