The question this answers
How much of my cross-region latency is a physical constant, and how much can I actually engineer away?
A hard lower bound, not an estimate: no message between two points can arrive faster than distance divided by the signal velocity in the medium. For standard single-mode fibre that velocity is roughly two-thirds of c, so any protocol requiring one round trip between London and New York costs at least ~56 ms, and every additional round trip costs at least another ~56 ms.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows when it sent a message and when a reply arrived. It cannot tell how that elapsed time divided between propagation, queueing, routing and remote processing. The floor below is the one part of the number it can compute in advance from a map — which makes it the only part it can reason about without measurement.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The arithmetic, done once, properly
In vacuum, light travels at 299,792 km/s. In single-mode fibre it does not: the refractive index of silica is about 1.47, so the group velocity is roughly c / 1.47 ≈ 204,000 km/s. Engineers round this to two-thirds of c, or about 200 km per millisecond, and that rounding is good enough for every decision you will make with it.
London to New York is about 5,585 km great-circle. Divide: 5,585 / 200,000 = 0.0279 s, so ~28 ms one way and ~56 ms round trip. That is the number before the first router, before TLS, before your service does anything. Measured RTT on a good path is 70–80 ms, so roughly 25–30% of what you observe is overhead you might influence, and 70–75% is a physical constant.
This ratio is the point. On a same-rack call, propagation is negligible and essentially all of the latency is software — so profiling pays. On an intercontinental call, propagation dominates and profiling the far service can at best recover the smaller share. The optimisation you reach for depends on which side of that line you are on, and most teams reach for the wrong one because they have never done this division.
signal velocity in single-mode fibre
c = 299,792 km/s
n (silica) ≈ 1.47
v = c / n ≈ 204,000 km/s → round to 200 km/ms
London → New York
great-circle distance 5,585 km
one-way floor 5585 / 200 = 27.9 ms
round-trip floor 55.9 ms
typical measured RTT 70 – 80 ms
------------------------------------------
physics ~56 ms (71 – 80% of observed)
everything else ~15-25 ms (routing, queueing, path
length, serialisation)The table you should keep
Do this once for the pairs you actually run between, and pin it somewhere the team sees it. The floor column is arithmetic; the measured column is what your provider’s backbone delivers today. The gap between them is your entire optimisation budget for that link — and it is smaller than people expect.
Read the Sydney row carefully. A single synchronous round trip between London and Sydney costs at minimum 170 ms and in practice around 250 ms. A two-phase commit — two round trips — costs half a second before any work happens, while holding locks the whole time. That is not a slow implementation of 2PC; that is 2PC working correctly at that distance, which is why [[two-phase-commit]] across continents is a design nobody defends twice.
| Pair | Distance | One-way floor | RTT floor | Typical measured RTT |
|---|---|---|---|---|
| London ↔ Amsterdamassumption | ~360 km | 1.8 ms | 3.6 ms | 7–10 ms |
| New York ↔ San Franciscoassumption | ~4,130 km | 20.6 ms | 41 ms | 60–70 ms |
| London ↔ New Yorkassumption | ~5,585 km | 27.9 ms | 56 ms | 70–80 ms |
| New York ↔ São Pauloassumption | ~7,680 km | 38.4 ms | 77 ms | 110–125 ms |
| Frankfurt ↔ Singaporeassumption | ~10,270 km | 51.4 ms | 103 ms | 150–180 ms |
| London ↔ Sydneyassumption | ~16,990 km | 85 ms | 170 ms | 240–280 ms |
Why the measured number is always higher
The floor assumes a straight fibre along the great circle. There is no such fibre. Cables follow cable landings, existing rights of way, seabed topography and the commercial history of whoever laid them, so the installed path is routinely 1.2–1.5× the great-circle distance. The London–New York fibre run is closer to 6,500–7,000 km than 5,585, which alone accounts for a good part of the gap.
Then add the things that are genuinely engineering: each router hop contributes forwarding delay and, under load, queueing delay — and queueing is the part that produces your p99 rather than your p50. Optical amplifiers and regenerators add small fixed amounts. Serialisation delay puts the packet on the wire. TLS costs one or two extra round trips on a cold connection, which at 75 ms each is why connection reuse matters far more across regions than within one. And a "region" is not a point: the datacentre may sit 50–100 km outside the city the region is named after.
Every one of those is, in principle, improvable. The 56 ms is not. That asymmetry is the whole lesson: you can shorten the queue, you cannot shorten the ocean.
- Installed fibre path ≈ 1.2–1.5× great-circle — the single largest contributor to the gap.
- Router forwarding and queueing — small at p50, dominant at p99, and the only part that varies with your own load.
- TLS handshake — 1–2 extra RTTs on a cold connection; at 75 ms per RTT this is a *150 ms* tax that connection pooling removes entirely.
- Serialisation and amplification — small, fixed, not worth your attention.
- Region-to-city offset — tens of km, but it is why measured never quite matches the map.
What this does to consensus
A consensus round is a round trip. In Raft, committing an entry means the leader replicates to followers and waits for a majority to acknowledge — one RTT to the *nearest majority*, not to the furthest node. That detail is the whole art of cross-region consensus placement.
Put a five-node group in London, London, Frankfurt, New York, New York with the leader in London. A majority is three: London, London, Frankfurt. The commit latency is the London↔Frankfurt RTT, about 15 ms — perfectly usable. Now move the leader to New York. Its nearest majority is New York, New York, and then either London or Frankfurt: commit latency becomes roughly 75 ms, a fivefold regression, caused by nothing but which node won an election. Leader placement is a latency decision that a failover can make for you at 3 a.m.
And the floor bites hardest where people least expect it: a linearizable read in a distant region. If the data’s leader is in London and a client in Singapore needs a read that cannot be stale, that read costs at least one London↔Singapore round trip — around 180 ms — no matter how much cache sits in Singapore. This is why [[linearizability]] across regions is a business decision, and why so many systems quietly offer [[session-guarantees]] instead.
The design conclusions that follow
Once you accept the floor, a short list of consequences falls out, and they are the reason this lesson anchors the module.
Count round trips, not milliseconds. Across a region boundary, latency is quantised in RTTs. A protocol that takes three round trips is three times as slow as one that takes one, and no implementation detail changes that ratio. This makes RTT-count the correct unit for reviewing a cross-region design.
Put the coordination where the majority is. If a group must agree synchronously, its members should be close to each other, even if that means they are far from some users. Spreading a quorum evenly across the globe for "balance" maximises exactly the wrong thing.
Move the decision, not the data. The cheapest cross-region write is the one that does not cross — which is [[multi-region-write-models]]’s partitioned ownership, and why it is so often the right answer.
And the honest caveat: there *are* faster media. Hollow-core fibre has a refractive index near 1 and gets much closer to c; microwave and laser links across land beat fibre because they go straighter and through air. High-frequency trading firms pay enormous sums for a few milliseconds on the Chicago–New York path this way. None of it is available to you as a cloud tenant, and none of it changes the ocean crossing much. Treat the floor as fixed.
Key points
- Light in fibre travels at about two-thirds of c — roughly 200 km per millisecond.
- London↔New York is ~5,600 km: a ~28 ms one-way floor and a ~56 ms round-trip floor, before any processing.
- Measured RTT is 70–80 ms, so roughly three-quarters of observed cross-Atlantic latency is physics and one quarter is engineering.
- Software cannot optimise this away. It can only reduce the number of round trips.
- A consensus round costs at least one RTT — to the nearest majority, which makes leader placement a latency decision.
- A linearizable read from a distant region costs a full RTT to the leader regardless of local caching.
- Across regions, review designs by counting round trips; that is the unit the physics is denominated in.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Take the great-circle distance between the two regions.
- • Divide by 200 km/ms to get the one-way propagation floor; double it for the round-trip floor.
- • Multiply the installed path by 1.2–1.5× if you want a realistic fibre-path floor rather than a theoretical one.
- • Count how many round trips your protocol needs on the critical path: one for a simple request, one for a Raft commit, two for 2PC prepare-and-commit, one or two more for a cold TLS handshake.
- • Multiply. The product is the latency you have committed to before writing any code.
- • A submarine cable cut reroutes traffic the long way round; RTT can double within seconds and stay doubled for weeks while a repair ship works.
- • A BGP change moves the path to a longer one; latency steps up permanently and no service reports an error.
- • Congestion on a shared link adds queueing delay, which shows up in p99 long before it shows up in p50.
- • The routing layer sends a client to a region that is not the nearest one, so the client pays a distance nobody designed for.
- • A cold connection pool forces a TLS handshake per request, adding 1–2 RTTs to every call — at cross-region distances this is the single most common self-inflicted latency wound.
- • A cross-region p99 that is a clean multiple of the RTT — say 230 ms on a 75 ms link — which tells the operator the protocol is doing three round trips, not that any component is slow. Nobody finds this by profiling CPU.
- • Latency doubles overnight with zero error-rate change and zero deploys: a cable cut or path change. The signal is RTT to the peer region, and most dashboards do not have it.
- • A failover moves the Raft leader to a distant region and write latency quintuples while every health check stays green. The operator sees "slow" with no failing component.
- • Timeout budgets set from single-region experience — say 100 ms — cause blanket failures the moment traffic crosses the Atlantic, because the deadline is below the physical floor for the round trips required.
- • Connection-pool exhaustion under cross-region load: each request holds a connection for 75 ms instead of 2 ms, so the same request rate needs ~35× the pool, and the observable is queueing at the client, not at the server.
- • Every synchronous agreement across a region boundary costs at least one RTT, and that cost is paid on every operation, not only during failures.
- • The relevant distance is to the nearest quorum, not the furthest member — so quorum placement is a latency lever, and often the only one available.
- • Adding a third region to a consensus group can *reduce* commit latency (a closer majority becomes reachable) or increase it (the group’s centre of mass moves away from the leader). It depends entirely on geography.
- • Coordination that can be moved off the critical path — asynchronous replication, deferred validation, escrowed budgets — buys back the whole RTT, which is why
[[coordination-avoidance]]is worth more here than anywhere else in the domain.
- • The floor does not change under failure; only the path length and the queueing do.
- • When a path is rerouted, protocols that assumed the old RTT begin timing out even though nothing is down — deadline budgets calibrated to a healthy path are themselves a failure mode.
- • Consensus continues to work at a longer RTT, just slower; the danger is election timeouts tuned for the short path now firing spuriously and causing leadership churn.
- • Detect: track RTT to each peer region continuously, with the computed floor drawn as a reference line so a path change is visible as a step, not lost in noise.
- • Contain: make deadlines proportional to the measured RTT of the path in use rather than a constant, so a rerouted path degrades latency instead of converting into errors.
- • Recover: nothing to recover — physics did not break. What you recover from is the *misconfiguration* the new RTT exposed, usually a timeout or an election timer.
- • Reconcile: re-derive timeout budgets and election timeouts from the current measured floor whenever regions are added or moved.
- • Verify: after any topology change, re-measure and compare against the floor. A ratio much above 1.5× means something in the path is worth investigating; near 1.0× means you are done, permanently.
- • Per-region-pair RTT at p50 and p99, with the great-circle floor as an annotation — the ratio is the only latency KPI here that means anything.
- • Round trips per operation on cross-region paths, derived from traces. This is the number you can actually change.
- • TLS handshake rate on cross-region connections; anything above near-zero means connection reuse is broken and you are paying double.
- • Consensus commit latency broken down by which node is currently leader — the metric that makes leader placement visible.
- • Distribution of client-to-region distance, to catch routing that sends users somewhere other than their nearest region.
- • Sizing a design before building it: the floor tells you in five minutes whether a synchronous cross-region write path can meet the latency requirement, which is usually the fastest "no" available.
- • Arbitrating between "the service is slow" and "the distance is long" during an incident.
- • Choosing quorum and leader placement, where the arithmetic gives an unambiguous answer.
- • Setting timeout budgets and election timeouts that survive a path change instead of causing one.
- • Within a single region, propagation is a rounding error and this reasoning distracts from the real cost, which is software.
- • For bulk transfer, latency is the wrong metric entirely — throughput and windowing dominate, and a high-latency link can move enormous data efficiently.
bandwidth-vs-latencyis the distinction. - • Used as fatalism: "it is physics" becomes an excuse for a protocol doing six avoidable round trips. The floor is a floor, not an alibi.
- • Remove the round trip rather than shorten it: serve from a local replica and accept staleness, which converts a latency problem into a consistency decision you can reason about.
- • Batch and pipeline so that N operations share one round trip instead of taking N — the single highest-leverage change on any cross-region path.
- • Move computation to the data (
[[data-locality]]) so that the boundary is crossed once with a result rather than repeatedly with queries. - • Give each region an exclusive slice of the keyspace so the common write never crosses at all —
[[multi-region-write-models]]. - • Accept asynchronous replication and design the reconciliation, trading a hard latency cost for a bounded staleness window and some
[[reconciliation]]work.
The floor under every cross-region design, in kilometres
signal velocity in single-mode fibre c = 299,792 km/s n (silica) ≈ 1.47 v = c / n ≈ 204,000 km/s → round to 200 km/ms London → New York great-circle distance 5,570 km installed path (× 1.00) 5,570 km one-way floor 27.9 ms round-trip floor 55.7 ms same trip at c, in vacuum 37.2 ms typical measured RTT 70–80 ms
What people believe, and what is true
A faster network will fix cross-region latency.
Bandwidth and propagation delay are unrelated. A 100 Gbps link between London and New York has exactly the same 56 ms floor as a 1 Gbps one; you can send more per round trip, never make the round trip shorter.
Caching solves it.
Caching solves it for data that may be stale. For a linearizable read or any write that enforces a global invariant, the round trip is mandatory and a local cache cannot help — that is precisely what the guarantee means.
The cloud provider’s private backbone avoids the physics.
A private backbone gives you a straighter path, less congestion and more predictable p99 — real and worth having. It buys back part of the 25% overhead. The 56 ms is untouched.
We measured 72 ms, so 56 ms of theory is not useful.
The 56 ms is what tells you 72 ms is close to optimal and that further tuning of that hop is wasted effort. A floor’s value is in telling you when to stop.
Adding regions reduces latency.
It reduces *client-to-region* latency for nearby users. If those regions join a synchronous quorum, it can simultaneously increase commit latency for everyone by moving the quorum’s centre of mass.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Light in fibre covers about 200 km per millisecond. London to New York is ~5,600 km, so a round trip cannot beat ~56 ms. Every synchronous cross-region step costs at least that, forever.
Practical
Compute the floor for your region pairs and put it on the dashboard beside the measured RTT. Then count round trips on the critical path — cold TLS, auth lookup, the write itself, a confirmation read — and multiply. Most cross-region latency complaints resolve to "four round trips where one would do", which is fixable, rather than "the ocean is wide", which is not.
Advanced
The floor turns latency into a structural property of a protocol: the RTT-count is fixed by the message pattern, so you can review a design for latency by reading its message diagram, before any code exists. This is also where the CAP/PACELC "else" clause becomes concrete — the interesting cost of strong consistency across regions is not the rare partition, it is the round trip you pay on every single operation for years. Systems that choose bounded staleness, session guarantees or escrowed local budgets are buying back RTTs, and the size of the prize is exactly the number in the table above.
Apply it
- 🔧 Build the floor table for the regions your system actually runs in, and mark which cross-region calls on your critical path exceed 1.5× their floor.
- 🔧 Take one cross-region request trace and count its round trips. Then design the version that uses one.
- ⚡ A team sets a uniform 100 ms RPC timeout that works fine in one region. Traffic is enabled to a second region across the Atlantic. Predict what happens and why the error looks like an outage rather than a latency regression.
- 💬 Estimate, from first principles, the minimum round-trip time between London and New York. Show your working.
- 💬 Your cross-Atlantic p99 is 240 ms on a link that measures 75 ms RTT. What does that tell you before you look at any code?
- 💬 Why does moving the Raft leader from London to New York change commit latency by 5×, and what would you do about it?
- 💬 A colleague proposes two-phase commit between Frankfurt and Singapore. Give the latency argument against it in one sentence.