The question this answers
What actually changes when my system spans more than one region?
None is added by going multi-region, and none is removed. The available consistency models are exactly the same as within a region; their latency cost rises by one to two orders of magnitude, and their availability now depends on a link you do not operate.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node in region EU knows what is in its own region: local disks, local peers, recent messages from US. It does not know whether US is up — only whether US answered recently. Within a zone that inference is usually safe because the failure is fast and rare; across a region boundary it is unsafe, because a fibre cut and a dead region look identical for minutes.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What a boundary costs, in numbers
A region is not a place on a map for design purposes. It is a latency step and a failure domain, and those two properties are the entire subject. Everything else — which cloud, which city, which product name — is Cloud’s material, and multi-region there covers it.
The step is what makes this a design decision rather than a deployment detail. Machines in the same rack agree in tens of microseconds. Machines in different availability zones agree in about a millisecond, which is why a synchronous three-zone quorum is a normal, boring choice. Machines in different regions agree in tens to hundreds of milliseconds, and a synchronous cross-region quorum on the write path is a decision somebody should have to defend in a design review.
The failure domain matters as much. Zones inside a region are designed to fail independently but share a region-level control plane, a region-level network fabric and often a region-level identity service. Regions share almost nothing — which is exactly why you want the second one, and exactly why the link between them is a component that fails on its own schedule.
| Boundary | Typical RTT | One synchronous round costs | What fails together |
|---|---|---|---|
| Same racktypical | 0.05–0.2 ms | Free relative to a disk write | Rack power, top-of-rack switch |
| Zone to zone, same regiontypical | 0.3–2 ms | Comparable to an fsync — invisible to users | Region control plane, region network fabric |
| Region to region, same continenttypical | 8–25 ms | Visible in p50; dominates a short transaction | Very little — usually only a shared provider control plane |
| Region to region, intercontinentaltypical | 70–280 ms | Dominates everything; two rounds exceed a human’s patience | Essentially nothing except the provider’s global services |
Four reasons to go multi-region, and why conflating them hurts
People say "we are going multi-region" as though it were one project. It is four different projects with four different designs, and the most common way to waste a year is to build one of them while believing you are getting all four.
Read latency for distant users needs replicas near users and tolerates stale reads. It does not need a second write path at all — a read-only replica and a CDN get most of it. Survival of a region loss needs a second copy of the data that is recoverable *and* a tested procedure for using it. Disaster recovery is the same data requirement with a much weaker time bound, expressed as rpo-and-rto. Legal residency needs data to be pinned, which is the opposite requirement to the first three — see [[data-residency]].
The conflation that costs most: teams build a warm standby for disaster recovery, then quietly begin routing read traffic to it for latency, then discover that the standby now has production load, production cost, and — because someone made it writable for one feature — production conflicts. The design that resulted was never chosen by anyone.
- Latency for users → replicas near users, stale reads acceptable, no new write path.
- Availability under region loss → a second usable copy plus a *rehearsed* failover, which is
[[region-active-passive]]. - Disaster recovery → a copy that survives corruption as well as outage; replication alone does not, because it replicates the corruption.
- Residency → the data must *not* leave, which forbids topologies the other three would choose.
The only question that matters: where is the write decided?
Reads spread easily. You can put a replica anywhere, and the only cost is staleness, which is a replication-lag problem the reader can often absorb. This is why "multi-region reads" is a weekend and "multi-region writes" is a year.
Writes are different because a write is where an invariant is enforced. "This username is unique", "this balance is not negative", "this seat is sold once" — each of those is a claim about *all* the data, and the moment two regions can both accept a write that touches the invariant, one of two things must be true: either they coordinate on the write path and pay the RTT from [[speed-of-light]], or the invariant is not actually enforced and you have chosen — perhaps without noticing — to detect and repair violations afterwards.
So the useful framing for the rest of this module is not "active-active or active-passive". It is: for each invariant, which region decides, and what happens when that region is unreachable? [[multi-region-write-models]] is that question answered three ways.
- EU (Frankfurt) — holds the write lease for `accounts`
- US (Virginia) — serves local reads, ~85 ms behind on p99
- AP (Singapore) — serves local reads, ~160 ms behind on p99
- apbelieves “my copy of `accounts` is current”✕ and it is false
- c1believes “the write I just made is visible to my next read”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
Two things a region boundary does that a zone boundary does not
First, the boundary is a metered link. Cross-region traffic is billed per byte, which turns a replication design into a cost design. Chatty protocols that are free inside a region — gossip, anti-entropy sweeps, full-object replication, verbose tracing — have a line item when they cross. Cloud’s egress-costs is the place for the pricing; the design consequence here is that cross-region protocols get engineered for *bytes*, and that pressure often pushes teams toward asynchronous, batched, lossy-under-failure replication for reasons that have nothing to do with correctness.
Second, the boundary is long enough to be partitioned without being down. Inside a region, a node that stops answering has usually stopped. Across regions, the link fails while both sides keep running and keep serving users — which is the precise condition that makes [[split-brain]] a real operational event rather than a textbook one. [[region-failure]] is that scenario in full.
Key points
- A region boundary changes two things only: the cost of a round trip, and what fails together. Everything else follows from those.
- Zone-to-zone agreement is ~1 ms and can sit on the write path. Region-to-region is 10–280 ms and generally cannot.
- "Multi-region" is four different projects — user latency, availability, disaster recovery, residency — with four different designs.
- Reads spread cheaply; writes do not, because a write is where an invariant is enforced.
- The design question is per invariant: which region decides, and what happens when it is unreachable?
- Across regions, "not answering" and "down" are indistinguishable for minutes, so partition handling stops being theoretical.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Choose the data’s home: which region is authoritative for each class of record.
- • Choose the replication mode to the other regions: synchronous (pay the RTT on every write) or asynchronous (accept a non-zero RPO).
- • Choose the read policy per call site: local-and-possibly-stale, read-your-writes via session tokens, or route to the home region and pay the RTT.
- • Choose the routing layer that maps a user to a region — DNS, anycast, or a global load balancer — and accept that it converges slowly and cannot be trusted as a fencing mechanism.
- • Choose what happens when a region is unreachable, per invariant, and write that down before the incident rather than during it.
- • The inter-region link degrades — packet loss and latency rise, but it does not go away, so failure detectors flap rather than fire.
- • One region loses its control plane while the data plane keeps serving, so you cannot deploy or scale there but traffic is still landing.
- • Replication falls behind under load, silently increasing RPO exactly when you are most likely to need it.
- • The global routing layer sends traffic to a region that is up but has stale data, producing correct-looking wrong answers.
- • A configuration change applied to one region only creates a version skew that replication carries into a schema error at the far side.
- • Silent staleness: users in the far region see their own write disappear on the next page load. Support tickets say "it did not save"; every service reports 100% success and no error appears anywhere.
- • Replication backlog: the cross-region lag metric climbs from 200 ms to 40 minutes over an afternoon under a bulk import. Nothing alerts because the write path is healthy; the exposure is only visible if you alert on lag itself.
- • Egress bill shock: cross-region traffic doubles after a replication change and the first signal is a finance question at month end, not a dashboard.
- • Flapping failover: a degraded (not dead) link causes health checks to oscillate, and an automated failover promotes and demotes a region repeatedly, each cycle dropping in-flight writes.
- • Asymmetric routing: the global load balancer moves users to region B while the write path still targets region A, so every request pays a full intercontinental RTT and p99 latency triples with no error rate change.
- • Reads require none, which is why they are cheap to distribute.
- • A write that enforces a global invariant requires agreement with whichever regions hold the deciding quorum — at minimum one inter-region RTT, and that RTT is a floor set by physics, not by tuning.
- • Failover requires agreement about *who is primary now*, and that agreement is the hardest one, because it must be reached exactly when the network that would carry it is broken.
- • Every cross-region coordination point converts an availability property into a joint availability property: the operation is now up only when both regions and the link between them are up.
- • Durability of already-committed local writes is unaffected — a region that committed does not un-commit because it lost contact.
- • Any invariant that spans regions is unenforced for the duration of the partition, whether or not you designed for that.
- • Read availability normally survives, because a stale local replica can still answer; the guarantee it offers silently drops from whatever you claimed to eventual.
- • Cross-region operations (a transfer, a global index update) queue, fail, or complete on one side only — the last being the one that produces reconciliation work.
- • Detect: alert on replication lag and on link RTT/loss, not only on service health. A healthy service with 40-minute lag is the dangerous state.
- • Contain: degrade explicitly — make far-region writes fail fast with a clear error rather than hanging for the full timeout budget, so the failure does not propagate into the caller’s thread pool.
- • Recover: bring the link back and let replication drain, watching lag return to baseline before restoring normal routing.
- • Reconcile: run the comparison the partition made necessary —
[[reconciliation]]and[[anti-entropy]]are the mechanisms; the fact that you need them is the price of the boundary. - • Verify: check the invariant, not the replication metric. Lag at zero means bytes arrived, not that the two sides agree about anything a user cares about.
- • Inter-region RTT and packet loss as a first-class dependency metric, with the floor from
[[speed-of-light]]drawn on the chart so degradation is visible against physics. - • Replication lag per region per stream, in seconds and in bytes — the second one predicts recovery time, the first one predicts data loss.
- • Fraction of writes that crossed a region boundary, broken down by call site. Rising means routing drifted, not that users moved.
- • Cross-region bytes per second by protocol, which is both the cost signal and the early warning for a chatty new dependency.
- • The reconciliation delta between regions for at least one business invariant, computed on a schedule and alerted on independently of everything above.
- • Users are genuinely spread across continents and the interaction is latency-sensitive — a 200 ms floor on every click is not something caching hides.
- • The business cannot accept the loss of a whole region, and has quantified that as an RTO the single-region design cannot meet.
- • Regulation requires data to be held in a specific jurisdiction, which is not a preference you can engineer around.
- • The workload is naturally partitionable by geography, so most requests never cross the boundary at all — the case where multi-region is cheap.
- • The system is not yet reliable within one region. A second region multiplies operational surface and will fail in the same ways, twice, less observably.
- • The dominant workload is a small number of strongly-consistent writes to shared state — you will pay the RTT on every one and gain nothing.
- • The team cannot rehearse a failover. An untested second region is a cost centre and a false sense of safety, not a redundancy.
- • The real complaint was p99 latency caused by a slow query. Geography does not fix a database problem; it hides it behind a bigger one.
- • Multi-zone within one region: most of the availability, roughly a thousandth of the coordination cost, and no conflict problem at all. This is the right answer far more often than it is chosen.
- • A CDN plus read replicas: solves distant-user *read* latency without a second write path. Architecture’s
cdn-architecturecovers the mechanism. - • Backup and restore to a second region with an honest RPO of hours, when the business can actually tolerate hours. Far cheaper and far more likely to work than an unexercised warm standby.
- • Move the users’ data to the users — per-tenant homing, which is
[[multi-region-write-models]]’s partitioned ownership, rather than making one dataset globally writable. - • Do nothing yet, and instead measure how much of the p99 is actually propagation delay. Frequently it is under a fifth.
What a region boundary actually changes, and what it does not
- EU (Frankfurt) — holds the write lease for `accounts`
- US (Virginia) — serves local reads, ~85 ms behind on p99
- AP (Singapore) — serves local reads, ~160 ms behind on p99
- apbelieves “my copy of `accounts` is current”✕ and it is false
- c1believes “the write I just made is visible to my next read”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
| Typical RTT | One synchronous round costs | What fails together | |
|---|---|---|---|
| Same racktypical | 0.05–0.2 ms | Free relative to a disk write | Rack power, top-of-rack switch |
| Zone to zone, same regiontypical | 0.3–2 ms | Comparable to an fsync — invisible to users | Region control plane, region network fabric |
| Region to region, same continenttypical | 8–25 ms | Visible in p50; dominates a short transaction | Very little — usually only a shared provider control plane |
| Region to region, intercontinentaltypical | 70–280 ms | Dominates everything; two rounds exceed a human’s patience | Essentially nothing except the provider’s global services |
What people believe, and what is true
Multi-region means the system survives a region outage.
It means a second copy exists. Survival requires a failover path that has been exercised recently enough to still work, which is a separate and much harder property.
Going multi-region improves latency.
It improves *read* latency for users near the new region. Write latency gets worse for everyone whose writes now cross a boundary, and that regression is easy to miss because it lands on a different percentile than the improvement.
The consistency model has to be weaker across regions.
It does not — you can run linearizable operations across regions. You simply pay the round trip on every one, and lose availability whenever the far side is unreachable. The model is a choice; only the price changed.
A second region doubles reliability.
It adds an independent failure domain *and* a new component — the link, the routing layer, the replication stream — each of which can take the system down on its own. Net reliability can go either way, and does.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
A region boundary makes agreement 10–100× more expensive and creates a link that fails while both sides stay alive. Decide, per invariant, which region gets to say yes.
Practical
Separate the four goals before designing: user latency, region-loss survival, disaster recovery, residency. Solve the cheapest one that is actually your problem. Then instrument replication lag and inter-region RTT as dependencies, because both fail silently while every service reports healthy.
Advanced
Treat the region boundary as a partition that is always about to happen, and enumerate invariants rather than services. For each invariant, the design space is: coordinate on the write path and lose availability under partition; give each region an exclusive slice so no coordination is needed; or allow divergence and define the merge. [[pacelc]] is the formalisation — CAP tells you what happens during the partition, the "else latency" half tells you what you pay the rest of the time, and across regions that second half is where the cost actually lives.
Apply it
- ⚡ A read replica is added in Singapore for latency. Two weeks later users there report that profile edits "sometimes revert". Explain what they are seeing and which single change fixes it.
- ⚡ Finance asks why the network bill tripled after a release that "only changed logging". Trace the plausible path.
- 💬 Your users are in Europe and North America and p99 checkout latency is 900 ms. Walk me through deciding whether a second region helps.
- 💬 What does adding a second region cost you that a third availability zone does not?
- 💬 A team says "we are multi-region so we are covered for a regional outage". What would you check before agreeing?