Multi-Region Systems
7 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
Crossing a region boundary multiplies the cost of agreement by roughly a hundred. Nothing else about the system changes — the same consistency models are available, the same protocols work. What changes is that every one of them now has a price you can feel, and a failure domain you cannot hide.
Q · What actually changes when my system spans more than one region?
London to New York is about 5,600 km. Light in fibre covers that in ~28 ms one way, ~56 ms there and back, before a single packet is routed, queued or processed. No framework, no protocol and no amount of tuning moves that number. It is the floor under every cross-region design decision.
Q · How much of my cross-region latency is a physical constant, and how much can I actually engineer away?
Single-writer region, multi-writer, or partitioned ownership where each region is authoritative for its own slice of the keys. The first is simple and slow for distant users; the second is fast and makes every conflict real; the third is the one people forget, and is very often the right answer.
Q · Which region is allowed to accept a write, and what does each answer cost me?
One region serves; another stands ready. The model is easy to explain and easy to be correct about — there is exactly one writer, so there are no conflicts, ever. Its two weaknesses are not conceptual: the failover is slow and multi-step, and it is almost never exercised, which means its probability of working the first time is far below what the runbook implies.
Q · A standby region exists. What actually happens when I have to use it, and how long has it been since anyone checked?
Both regions serve, both accept writes, there is no failover step to get wrong. What you buy is local latency and continuously-proven capacity. What you pay is that every concurrent-write scenario you could previously wave away is now a thing that happens, in production, at a rate you do not control — and that a bad write propagates to both regions at wire speed.
Q · Both regions are serving live traffic. Which problems did that solve, and which did it create?
The link between two regions fails. Both are healthy, both are serving users, neither can reach the other, and neither can tell whether the other is dead or merely unreachable. Whether both may keep accepting writes has an answer — but it is a property of the invariant, not of your preference, and it is different for different data in the same system.
Q · The regions cannot see each other and both are up. Which of them is allowed to say yes?
Some data must legally remain inside a jurisdiction. That is not a deployment preference — it is a hard constraint that propagates upward into your partitioning key, your replication topology, your quorum placement, your indexes, your caches, your backups and your logs. It can forbid outright the design you would otherwise have chosen, and it is best discovered before you build it.
Q · A class of my data may not leave its jurisdiction. What does that forbid, and what does it force?