Multi-Region Systems

Three Ways to Accept a Write in More Than One Place

Single-writer region, multi-writer, or partitioned ownership where each region is authoritative for its own slice of the keys. The first is simple and slow for distant users; the second is fast and makes every conflict real; the third is the one people forget, and is very often the right answer.

▶ Run the lab

The question this answers

The question

Which region is allowed to accept a write, and what does each answer cost me?

The guarantee — the property claimed, and its scope

Single-writer: linearizable for all keys, at the cost of a full RTT for distant writers and unavailability of writes when the writer region is unreachable. Multi-writer: local write latency everywhere, with a ceiling of causal or eventual consistency and no cross-region invariant. Partitioned ownership: linearizable per key with local latency for keys owned locally — and no atomicity across partitions without explicit coordination.

Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.

What a node knows — observation versus inference

A region knows which keys it currently believes it owns, and when it last heard from the others. It does not know whether another region has been granted ownership of a key it thinks is its own — an ownership record it has not yet seen is indistinguishable from one that does not exist. Every one of the three models is, underneath, a different answer to "how does a region know it is still allowed to say yes?"

A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.

What guarantee?What does a node know?How does it work?What can fail?How does it fail?Where is coordination?What holds under failure?How does it recover?How would you know?What is the simpler thing?
multi-regionwrite pathownershipconflictspartitioning

The comparison, made concrete

Assume three regions — Frankfurt, Virginia, Singapore — with the RTT floors from [[speed-of-light]]: FRA↔IAD about 90 ms, FRA↔SIN about 160 ms, IAD↔SIN about 210 ms. Now put a write in each model and follow it.

Read the table as four independent questions, because they fail independently. A model can have excellent write latency and terrible failover. A model can have no conflicts and still lose data. Teams routinely choose on the latency column alone and are then surprised by the failover column eighteen months later, during the incident that made them look.

ModelWrite latencyFailover behaviourConflict potentialConsistency achievable
Single-writer regionprotocolLocal writers: ~2 ms. Distant writers: one full RTT, 90–210 ms, on *every* write.Hard. Promoting a new writer needs agreement about who is primary, plus fencing. Minutes if rehearsed, hours if not. RPO > 0 under async replication.None by construction — there is only ever one place a write is ordered.Linearizable for every key. The strongest option, and the only one that gives you global uniqueness for free.
Multi-writer (multi-leader)protocolAlways local: ~2 ms everywhere. This is the whole appeal and it is a real one.Trivial — the surviving regions never stopped accepting writes. There is nothing to promote.Maximal. Any two regions writing the same key concurrently conflict, and "concurrently" means within the replication lag window, which is 100 ms to minutes.Causal at best, eventual in practice. No global uniqueness, no cross-region invariant, no read-modify-write that is safe without extra machinery.
Partitioned ownershipprotocolLocal for keys the region owns — the common case if homing is chosen well. One RTT for foreign keys, same as single-writer.Per-partition. Only the slices owned by the lost region need re-homing, so the blast radius is a fraction of the data and the operation is repeatable and small.None *within* a partition. The residual problem is operations spanning two partitions, which is a smaller and better-understood problem.Linearizable per key. No atomicity across partitions without an explicit protocol — a saga or a coordinated transfer.
The three write models against the four properties that matter

Single-writer: the honest baseline

One region holds the write lease for the dataset. Everyone else reads locally and forwards writes. This is [[leader-based-replication]] with the leader’s replicas placed across an ocean, and it is the model to beat, because it is the only one where you can state a guarantee in one sentence and have it be true.

Its cost is precisely legible: a user in Singapore writing to a Frankfurt primary waits 160 ms plus processing, on every write, forever. If the interaction is one write per user action, that is fine and nobody complains. If the interaction is a form that autosaves per keystroke, it is unusable, and the fix is not "make the network faster" — it is a different write model or a different interaction.

The failure story is where single-writer earns its complexity. When the writer region is unreachable, writes stop everywhere. That is not a bug; it is the guarantee working — the system is refusing to let a second region create a divergent history. Whether you *want* that refusal is the CAP choice, made concrete, and [[cap-theorem]] is the framing. What you must not do is add "temporary" write capability elsewhere during an incident, because that converts a clean unavailability into a permanent divergence that nobody will reconcile correctly.

Multi-writer: fast, and every conflict becomes your problem

Every region accepts writes locally and ships them to the others asynchronously. Write latency is ~2 ms in all regions, region loss requires no failover step, and the model is genuinely the right choice for some data. The bill arrives on the read side and the correctness side.

The conflict window is the replication lag, not "the same instant". Two edits 300 ms apart in different regions are concurrent for conflict purposes if replication takes 400 ms — and users have no intuition for this, so the bug reports say "it overwrote my change" and the logs say everything succeeded. [[write-conflicts]] is the general treatment; the geo-specific point is that the window is *large* and *variable*, and grows exactly when the system is under stress.

What multi-writer cannot do is enforce an invariant that spans regions. A globally unique username, a non-negative balance, an inventory of one hundred seats: none of these survive two regions both saying yes. Systems that appear to manage it are doing one of three things — restricting the invariant to a single owner (which is the third model), pre-splitting the resource so no coordination is needed (escrow, and see [[coordination-avoidance]]), or detecting the violation afterwards and repairing it, which means someone gets a "your username has been changed" email.

Choose multi-writer when the data is naturally append-only, or when a merge function genuinely exists — a shopping cart that unions, a counter that is a [[crdts]]-style sum, a document with operational merge, a set of tags. Choose it deliberately for those, not as a default because the latency column looked good.

Multi-writer: both regions said yes, and both were right locallysimplified
fra ↔ iad: slowfra ↔ sin: okiad ↔ sin: okFrankfurt · leader · up — accepted `set nickname = "kim"` at t=0Frankfurt★ leaderVirginia · leader · up — accepted `set nickname = "kimberly"` at t=0.2sVirginia★ leaderSingapore · follower · up — will receive both, in an order neither region choseSingapore· followerslow
slowok
  • Frankfurt — accepted `set nickname = "kim"` at t=0
  • Virginia — accepted `set nickname = "kimberly"` at t=0.2s
  • Singapore — will receive both, in an order neither region chose
What each node believes
  • frabelieves “the nickname is now "kim"”✕ and it is false
  • iadbelieves “the nickname is now "kimberly"”✕ and it is false
  • sinbelieves “there is a single agreed nickname”✕ and it is false

Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.

Partitioned ownership: the underrated answer

Split the keyspace and give each region exclusive authority over its slice. Frankfurt owns the EU tenants; Virginia owns the Americas; Singapore owns APAC. A write to a key lands in its owning region — locally if the user is where their data is, forwarded if not. Within a partition you have single-writer semantics with all of its guarantees; across the system you have local latency for the overwhelming majority of writes.

This is not a compromise between the other two. It is a recognition that most data has a natural home. A user’s account, a tenant’s workspace, an order and its history, a device’s telemetry — these are almost never written by two continents at once. The multi-writer conflict problem is largely an artefact of pretending that a globally-shared dataset exists when what you actually have is millions of small, independently-owned datasets that happened to be stored in one table.

The mechanics are those of [[why-partition]], applied with geography as the partition key’s home rather than as the key itself. You need three things. A directory: something that maps a key to its owning region, replicated everywhere and read on every request — usually cached hard, because a stale directory entry sends a write to the wrong place. An ownership handoff protocol: a user relocates, a region is lost, load rebalances, and a partition must change hands. This is a lease with a fencing token, exactly as in [[leases]] and [[fencing-tokens]], and getting it wrong is how you get two regions believing they own the same key. A cross-partition story: transferring funds between an EU account and a US account touches two owners, and there is no free atomicity there — it is [[cross-partition-operations]] and usually a saga.

The honest costs: the directory is a new globally-consistent thing to run, and it is on the critical path. Homing can be wrong — a user who travels, a tenant with staff on three continents, a "global" admin view that touches every partition. And rebalancing is a real operation with real risk, because it moves authority, not just bytes. But compare that list against "design a merge function for every field in the system and explain to users why their edit vanished", and partitioned ownership looks a great deal better than its reputation.

  • Ownership follows the *entity*, not the request. A German user’s data lives in Frankfurt even when they are travelling in Tokyo.
  • The directory is the coordination point, and it changes rarely — so it can be aggressively cached with a lease, unlike the data itself.
  • Ownership changes need fencing. A region that lost ownership but has not noticed will keep accepting writes; only a token the storage layer checks stops it.
  • Cross-partition operations are the residual problem, and they are a minority of traffic — which is the whole point of choosing the partition well.
  • A "global" read that spans partitions is a scatter-gather with [[fan-out-tail-latency]] behaviour, and is usually better served by an asynchronously-built [[materialized-views]].

Choose per invariant, not per system

The final move is to stop asking "what write model does our system use?" A real system uses several, and the split is by invariant.

Billing ledgers want single-writer, because the invariant is arithmetic and a conflict is a financial discrepancy. User profiles want partitioned ownership, because each profile has an obvious owner. Feature-flag configuration wants single-writer with global read replicas, because it is written rarely and read constantly. Activity feeds and audit logs want multi-writer, because they are append-only and merging is trivial. Session state wants regional ownership with no cross-region replication at all — losing it on a region failure is acceptable and much cheaper than replicating it.

Written out that way, the design becomes reviewable. Any table in your system should be able to answer: which region decides, what happens if that region is unreachable, and what merges if two writes race. If a table cannot answer all three, that table is where your next multi-region incident will start.

Key points

  • Single-writer gives linearizability for every key and makes distant writers pay a full RTT on every write.
  • Multi-writer gives local write latency everywhere and makes every conflict scenario real and permanent.
  • Partitioned ownership gives single-writer guarantees per key plus local latency for well-homed keys — and is the option most often skipped.
  • The conflict window in multi-writer is the replication lag, which is large, variable, and worst under load.
  • No multi-writer design enforces a cross-region invariant. Systems that appear to are either single-owner underneath, escrowed, or repairing violations after the fact.
  • Partitioned ownership costs a directory, an ownership handoff protocol with fencing, and an explicit story for cross-partition operations.
  • Choose the model per invariant. One system will legitimately use all three.

The chain, answered

Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.

How it works
  • Enumerate the invariants, not the services: uniqueness, non-negativity, at-most-once allocation, monotonic sequence, free-form text.
  • For each, ask whether two regions accepting writes simultaneously could violate it. If yes, that data needs a single owner.
  • For data needing a single owner, ask whether the owner can be per-key (partitioned ownership) rather than per-dataset (single-writer). It usually can.
  • Publish the ownership directory to every region and cache it under a lease, so the common path is a local lookup.
  • Route each write to its owner: local if owned here, forwarded and awaiting the RTT if not.
  • Give the owner a fenced lease so that a region which has lost ownership cannot commit, even if it has not yet learned it lost it.
  • For the remaining genuinely-shared data, either define a merge function and accept eventual consistency, or accept the RTT.
What can fail at the boundary
  • The ownership directory is stale in one region, which forwards writes to a region that no longer owns the key.
  • Two regions both believe they own a partition during a handoff, and both accept writes.
  • The forwarding path to a foreign owner times out, and the caller cannot tell whether the remote write committed — [[timeout-ambiguity]] at a 200 ms scale, so it happens far more often.
  • Replication between multi-writer regions falls behind, silently widening the conflict window from milliseconds to minutes.
  • A user or tenant is homed in the wrong region, so every one of their writes crosses an ocean and their experience is uniformly bad while the fleet-wide metrics look fine.
How it fails — what an operator sees
  • Lost update in multi-writer: a user edits a profile in Frankfurt, a support agent edits it in Virginia, last-write-wins discards one. The operator sees no error at all — the only evidence is a support ticket saying "my change is gone".
  • Duplicate unique value: two regions each accept a signup for the same email address. The observable is a uniqueness constraint violation appearing in the *replication stream* hours later, or two accounts that both work until someone tries to log in.
  • Split ownership: after a handoff, two regions accept writes to the same key. The operator sees the row alternating between two values in a read-heavy dashboard, and each region’s logs look internally consistent.
  • Mis-homed tenant: one customer’s p99 is 400 ms while the global p99 is 40 ms. Nothing alerts, because per-tenant latency is not on any dashboard; the signal arrives as a churn risk from the account team.
  • Forwarding storm: the owning region slows down, every other region’s forwarded writes queue and retry, and the owner’s load doubles from retries alone — [[retry-amplification]] with an RTT-sized retry interval.
Where coordination is required
  • Single-writer: coordination on every distant write (one RTT) and heavy coordination on failover (agreement about the new primary).
  • Multi-writer: no coordination on the write path at all, which is the entire point, and unbounded coordination *afterwards* in the form of conflict resolution and reconciliation.
  • Partitioned ownership: coordination only on ownership changes, which are rare. This is the trade that makes it attractive — it moves the agreement from the request path to the topology-change path, which happens thousands of times less often.
  • Cross-partition operations reintroduce coordination in proportion to how badly the partitioning matches the workload — which is the real test of a homing scheme.
What still holds under failure
  • Single-writer: writes are unavailable when the writer is unreachable; reads continue and are stale. The invariant is preserved throughout.
  • Multi-writer: everything stays available and the invariant is simply not enforced during the partition. Divergence accumulates at the write rate.
  • Partitioned ownership: only the partitions owned by the lost region are unavailable for writes. Everything else is unaffected — the smallest blast radius of the three.
  • In all three, already-committed data remains durable; what varies is what a region is permitted to add to it.
How it recovers
  • Detect: compare the ownership directory against what each region believes it owns; disagreement is the signal that matters and it is cheap to compute.
  • Contain: on a suspected split ownership, revoke rather than reassign — stop both sides accepting writes until the lease is re-established with a higher fencing token.
  • Recover: for a lost region under partitioned ownership, re-home its partitions one at a time with fencing, which is a routine operation you can rehearse without an outage.
  • Reconcile: for multi-writer, run the merge deliberately rather than letting last-write-wins run it for you — surface the conflict to the application or the user where the value justifies it.
  • Verify: assert the invariant directly after recovery. Replication being caught up says nothing about whether two regions each created a "unique" row.
How you would know
  • Fraction of writes served by the local region versus forwarded — the single best health metric for a homing scheme, and it drifts silently as users move.
  • Conflict rate per key class in multi-writer, and the resolution taken. A zero conflict rate usually means you are not detecting them, not that they do not happen.
  • Ownership directory staleness: the age of the newest directory entry each region has seen.
  • Fencing-token rejections at the storage layer. Non-zero is normal during handoffs; sustained non-zero means a region has not accepted that it lost ownership.
  • Per-tenant p99 write latency, not just fleet p99 — mis-homing is invisible in the aggregate.
When it helps
  • Single-writer: when the invariant is strict, writes are relatively rare, and users are concentrated near one region.
  • Multi-writer: when the data is append-only or has a genuine merge function, and availability under partition is worth more than a strict invariant.
  • Partitioned ownership: when the data has a natural owner — per user, per tenant, per device — which is the overwhelming majority of application data.
  • Any of them: when the choice is made per invariant and documented, so the next engineer does not have to reverse-engineer it from an incident.
When it hurts
  • Single-writer when most users are far from the writer: you have built a system whose latency is set by an accident of where the database was first deployed.
  • Multi-writer for anything with a uniqueness or balance invariant: you have chosen to detect violations rather than prevent them, whether or not anyone said so out loud.
  • Partitioned ownership when the workload genuinely spans partitions — a global leaderboard, a cross-tenant analytic — because then most operations pay the coordination you designed to avoid.
  • All three when the real problem is that a single region was never made reliable. The write model does not fix an unreliable primary; it gives it a bigger stage.
Simpler alternatives
  • Single region for writes, replicas everywhere for reads, plus [[session-guarantees]] so users see their own writes. Solves most perceived multi-region problems at a fraction of the cost.
  • Escrow the constrained resource: pre-allocate a slice of the inventory or budget to each region so each can spend its own share without asking — [[coordination-avoidance]].
  • Make the contested data a CRDT so concurrent writes merge by construction and the model question stops mattering for that field — [[crdts]].
  • Queue distant writes and acknowledge asynchronously, converting a latency problem into a job-status problem the UI can express honestly.
  • Do not replicate at all: some data (sessions, caches, drafts) is cheaper to lose and recreate than to make globally consistent.

Which region is allowed to say yes, and what each answer costs

Three ways to accept a write in more than one place
Frankfurt, Virginia and Singapore. FRA↔IAD about 90 ms, FRA↔SIN about 160 ms, IAD↔SIN about 210 ms. Which region is allowed to say yes?
write model
the writer region
where the user is
write latency for this user
162 ms
conflict window
none — one ordering point
failover
promote + fence, whole dataset
consistency ceiling
linearizable per key
the invariant this data has to hold
This model can enforce it. Uniqueness is a claim about absence, and absence is what a region writing alone cannot verify. The price is visible above: a user in Singapore waits 160 ms plus processing on every write, forever.
R + W = 4 > N = 3. Quorums overlap on 1 node. The price: writes survive only 1 node failure and reads only 1. Overlap is bought with availability, not with cleverness.
Choose the model per invariant. One system will legitimately use all three. Billing ledgers want single-writer. User profiles want partitioned ownership. Feature-flag configuration wants single-writer with global read replicas. Activity feeds and audit logs want multi-writer. Session state wants regional ownership with no cross-region replication at all. For each table, answer three questions: which region decides, what happens if that region is unreachable, and what merges if two writes race.
Write latencyFailover behaviourConflict potentialConsistency achievable
Single-writer regionprotocolLocal writers: ~2 ms. Distant writers: one full RTT, 90–210 ms, on every write.Hard. Promoting a new writer needs agreement about who is primary, plus fencing. Minutes if rehearsed, hours if not. RPO > 0 under async replication.None by construction — there is only ever one place a write is ordered.Linearizable for every key. The only model that gives you global uniqueness for free.
Multi-writer (multi-leader)protocolAlways local: ~2 ms everywhere. This is the whole appeal and it is a real one.Trivial — the surviving regions never stopped accepting writes. There is nothing to promote.Maximal. Any two regions writing the same key concurrently conflict, and "concurrently" means within the replication lag window, which is 100 ms to minutes.Causal at best, eventual in practice. No global uniqueness, no cross-region invariant, no safe read-modify-write without extra machinery.
Partitioned ownershipprotocolLocal for keys the region owns — the common case if homing is chosen well. One RTT for foreign keys.Per-partition. Only the slices owned by the lost region need re-homing, so the blast radius is a fraction of the data.None within a partition. The residual problem is operations spanning two partitions.Linearizable per key. No atomicity across partitions without an explicit protocol — a saga or a coordinated transfer.
The three write models against the properties that matter. Read them as independent questions, because they fail independently.
typicalWorking RTT figures for those region pairs, not floors — substitute your own measurements before deciding on them. The comparison also treats each model as pure; real deployments mix them, and the sharpest bugs live in a transaction that touches two.

What people believe, and what is true

Claim

Multi-writer is the natural evolution of single-writer once you outgrow one region.

Reality

They are not points on a line. Multi-writer changes the guarantee, not the scale. Partitioned ownership is the actual next step for most systems, and it keeps the guarantee.

Claim

Conflicts are rare, so last-write-wins is fine.

Reality

The conflict window is the replication lag, which grows under exactly the load that also increases write volume. Conflicts cluster in incidents, which is the worst time for silent data loss.

Claim

Partitioned ownership is just sharding.

Reality

Sharding distributes storage; partitioned ownership distributes *authority*. The hard parts are the directory, the fenced handoff and cross-partition operations, none of which appear in a same-region sharding design.

Claim

The database handles conflicts for us.

Reality

It applies a resolution policy — usually last-write-wins by timestamp, which depends on clocks you do not control, see [[clock-skew]]. That is a policy, not a solution, and it discards data by design.

Claim

A user always writes from the region nearest them.

Reality

They write from wherever they are, which changes. Homing is a property of the data, not the request, and the two diverge for travellers, VPN users, admin tools and background jobs.

Go deeper

Only the levels this lesson can honestly fill — a missing level is a claim nobody had.

Overview

Pick one: one region decides (strong, slow for the far side), every region decides (fast, conflicts are real), or each region decides for its own keys (strong per key, fast for well-homed data). The third is usually right.

Practical

List your invariants and mark which could be violated by two regions saying yes at once. Those need an owner. Then check whether the owner can be per-key rather than per-dataset — per user, per tenant, per device. Build the directory, fence the ownership, and route writes to the owner. Measure the local-versus-forwarded write ratio; if it is below about 90%, your homing is wrong, not your architecture.

Advanced

The three models are the same protocol with different partitioning of *authority over an invariant*. Single-writer puts one authority over everything; multi-writer puts no authority anywhere and substitutes a merge; partitioned ownership factors the invariant so each region holds a disjoint piece of it. Seen that way, escrow is just partitioned ownership applied to a numeric resource — the "100 seats" invariant factored into "40 seats here, 60 there", each locally enforceable. That factoring is the general technique, and where it exists it beats both alternatives, because it buys single-writer correctness at multi-writer latency. Where it does not exist — global uniqueness over an unbounded namespace, a strict global sequence — you are back to paying the round trip, and [[coordination-cost]] tells you what that costs in availability as well as latency.

Apply it

Build it, then break it
  • 🔧 Take one table in your system and answer the three questions: which region decides, what happens if it is unreachable, what merges if two writes race.
  • 🔧 Design the ownership handoff for a tenant relocating from Frankfurt to Virginia, including what stops the old owner committing after the handoff.
Reason about this
  • Marketing runs a global campaign; signups spike in all three regions at once on a multi-writer store with a unique-username constraint. Describe the sequence of events over the next six hours.
Interview questions
  • 💬 You need to accept writes in Europe and North America. Walk me through the three models and which invariants decide between them.
  • 💬 What exactly does partitioned ownership need that plain sharding does not?
  • 💬 A multi-writer setup has a uniqueness constraint on email. What happens, and when do you find out?
  • 💬 Your local-write ratio is 60%. What does that tell you and what would you change?