Operating in More Than One Region
Two regions is not two copies of one system. Deploys, migrations, config, secrets and data all become distributed problems you now operate every day.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does running in a second region cost operationally, every day, in exchange for surviving the loss of one?
Multi-region is adopted for availability and paid for in daily operations. The permanent tax — every change now has an ordering problem — is rarely part of the decision.
Deploy the same stack to a second region, replicate the database, and put a global load balancer in front. Now we survive a region failure.
Schema migrations become a distributed change: a migration applied in one region reaches the other through replication, and code that has not shipped there yet must tolerate it (Expand, Migrate, Contract).
- Schema migrations become a distributed change: a migration applied in one region reaches the other through replication, and code that has not shipped there yet must tolerate it (Expand, Migrate, Contract).
- Configuration and feature flags drift between regions, producing behaviour that reproduces in one place and not the other — the most expensive kind of bug to chase.
- Data that is writable in more than one place needs a conflict story, and "we will not have conflicts" is not one.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The choice that determines everything else is where writes are accepted: a single writable region with read replicas elsewhere, or multiple writable regions.
- Single-writer keeps the data model simple and pays cross-region write latency for users far from the writer. Multi-writer removes that latency and introduces conflict resolution as a permanent application concern.
- Whatever the topology, changes now have an ordering requirement. Two regions running different code against replicated data is exactly the version-coexistence problem, with a network in the middle and a longer window.
- The failover path is only maintained if it is used. Serving real traffic from both regions keeps the path warm; a standby that never serves is a hypothesis (Region Failover).
Where writes go decides everything else
This is the first decision and the one that is hardest to reverse. Everything downstream — migrations, conflict handling, failover complexity, write latency — follows from it.
Which regions may accept writes for this dataset?
when Writes are a minority of traffic, or correctness under conflict matters more than write latency for distant users.
cost Distant users pay cross-region latency on writes. Losing the writer region means a promotion step, with the replication gap as data loss.
when Records naturally belong to one region — a customer, a tenant, a jurisdiction — and cross-partition writes are rare.
cost Routing must always agree on which region owns a record, and the rare cross-partition operation becomes a distributed transaction problem. Data residency rules often make this shape mandatory rather than optional.
when Low write latency everywhere is a product requirement and the data model can express merges or last-writer-wins honestly.
cost Conflict resolution becomes permanent application logic, and silent loss is the default failure mode when it is wrong. The hardest of the three to operate and to reason about.
when A region-wide outage is survivable as downtime, and drilled backups plus a rebuild meet the objectives.
cost Recovery time bounded by restore and rebuild rather than by a traffic shift. Frequently the correct answer, and rarely the one that gets proposed.
The daily operational tax
Each row is something that was one operation and is now two with an ordering constraint. None of them is hard on its own; together they are the reason multi-region is an operating model rather than a configuration.
| Operation | Single region | Multi-region | What bites |
|---|---|---|---|
| Deploy | One rollout, one verification | Ordered rollouts with a gate between | A long window of mixed versions against shared data |
| Schema migration | Apply, then deploy | Expand everywhere, deploy everywhere, contract last | Replication carries the schema change to older code |
| Config change | One source, one target | One source, rendered per region, drift-checked | Silent divergence; bugs that reproduce in one region only |
| Secret rotation | Rotate and restart | Rotate, replicate, restart in order | A region left with the old credential fails at startup |
| Alerting | Service-level thresholds | Region-dimensioned thresholds and rollups | Aggregates hide a fully failed region until it is half the traffic |
| Capacity | Size for peak plus headroom | Size each for peak plus the other region's load | Failover saturates a target sized for its own share |
The shared parts are the ones that take both regions down
Regions are independent failure domains only to the extent that nothing is shared. In practice every stack has shared components, and the useful exercise is to enumerate them rather than to pretend they are absent.
- The five nodes at the top are shared. Each is a way for both regions to fail at once, and each is a legitimate design choice that should be a named, accepted risk.
How to do it properly
Most important first.
- Decide and document the write topology per dataset. Different datasets in the same product legitimately make different choices.
- Deploy through one pipeline that treats regions as ordered stages, with verification between them — the second region is a natural canary boundary (Promotion).
- Make configuration and flags region-aware by design: a single source of truth, rendered per region, with drift detection rather than manual parity (Configuration Drift).
- Run migrations expand-first and confirm every region is running compatible code before the contract step.
- Serve real traffic from every region you intend to fail over to, and hold enough headroom in each that one can take the whole load.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Region-scoped changes and region-ordered deploys contain most of it. What is not contained is anything shared across regions — a global control plane, a shared config source, a single identity provider.
What can go wrong
- A migration applied in one region breaks the other region's older code through replication.
- A shared global component fails and both regions go down together, which is the exact scenario multi-region was bought to prevent.
- Conflict resolution in a multi-writer setup silently drops one side of a conflict and the loss is only found through customer reports.
- "Two regions doubles availability." It changes which failures you survive, and adds coordination failures that a single region cannot have.
- "Replication makes the second region equivalent." Asynchronous replication makes it *nearly* equivalent, and the gap is exactly your data loss on promotion.
- "We are multi-region, so we do not need backups." Multi-region protects against losing a region. It replicates a bad migration perfectly (Backup Operations).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every dashboard and alert carries a region dimension, and a single-region failure is visible as such.
- Replication lag per region is a monitored signal with a threshold tied to the stated recovery point.
- Each region has served the full production load at least once, in a rehearsal or a real event.
- Rolling back a change means rolling it back everywhere, in the reverse order of deployment, and the intermediate state is again mixed versions.
- Retreating from multi-region to single-region is a real and legitimate option when the operational cost exceeds the availability benefit. It is much easier to do deliberately than to discover you are effectively single-region because the second one has drifted.
- Automate: region-ordered deploys with verification gates, config rendering from one source, drift detection, replication lag alerting, and per-region health rollups.
- Keep human: the write-topology decision, conflict resolution policy, and the choice to run degraded in one region rather than fail over.
- Availability improves for infrastructure loss and operational complexity increases for every change, permanently. Most teams underestimate the second term.
- Active/active removes the failover step and demands the hardest data model. Active/passive keeps the data model simple and keeps a failover procedure that must be rehearsed.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- CLOUD-SPECIFICWhich services are global, regional or zonal differs per provider, and mixing them determines your real failure domains. A global service can fail globally; a regional one cannot, but also cannot help another region. Read the scope of every managed service you depend on rather than assuming symmetry.
- SCALE-SPECIFICBelow the point where a region-wide outage would be existential, single-region with drilled backups and a documented rebuild is usually the better engineering choice. Multi-region has a fixed operational overhead that a small team pays every day.
- DATABASE-SPECIFICCross-region data options vary from async read replicas, to promotable standbys, to genuinely multi-writer systems with conflict resolution. Each imposes a different application contract, and the operational burden differs more than the marketing does.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — consistency models, replication topologies and conflict resolution: what the data layer can actually promise across regions, and at what latency cost.