Cross-Region Latency Is Physics, Not Configuration
Light in fibre travels about 200,000 km/s. Frankfurt to Virginia and back is roughly 13,000 km of that, so no amount of tuning gets a round trip under about 65ms — and a request that crosses the Atlantic four times has spent a quarter of a second before doing any work.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The floor you cannot optimise past
Signal propagation in optical fibre runs at roughly two thirds the speed of light in vacuum — about 200,000 km/s. That gives a hard lower bound on round-trip time between any two points on Earth, and it is not a bound that better hardware, a faster runtime or a bigger instance can move.
The numbers below derive the theoretical floor from great-circle distance, then note what is typically observed. Real fibre routes do not follow great circles — they follow cable landing points, existing rights of way and commercial peering arrangements — so actual paths commonly run 1.3 to 1.6 times the straight-line distance, plus switching and queueing at every hop. Observed RTT is therefore reliably worse than the floor, never better.
Treating this as a fixed cost rather than a problem to solve changes the engineering conversation productively. You cannot make the Atlantic narrower; you can decide how many times a request crosses it, which is entirely within your control.
| Route | Great-circle distance | Theoretical RTT floor | Typically observed |
|---|---|---|---|
| Same rack | < 0.1 km | ≈ 0 ms | Well under 1 ms |
| Same region, cross-AZ | tens of km | < 1 ms | Typically low single-digit ms |
| London ↔ New York | ≈ 5,570 km | ≈ 56 ms | Commonly ~70–80 ms |
| Frankfurt ↔ N. Virginia | ≈ 6,600 km | ≈ 66 ms | Commonly ~85–95 ms |
| London ↔ Sydney | ≈ 17,000 km | ≈ 170 ms | Commonly ~250 ms or more |
The ping-pong anti-pattern
The expensive mistake is not being multi-region. It is being multi-region *incoherently*: a request that enters in Europe, calls an API in the US, which reads a database in Europe, then calls a dependency in the US, has crossed the Atlantic four times to serve one user. At ~90ms per crossing that is 360ms of pure propagation before any service does a millisecond of work.
This shape almost never results from a decision. It accretes: the API was deployed close to the team, the database stayed where the data-residency requirement put it, the payment provider has a US endpoint, and each individual choice was locally reasonable. Nobody drew the resulting path until users complained.
The fix is data locality rather than service locality. Putting a stateless service in every region is easy and buys nothing if it still reaches across an ocean for its data on every request. The hard and valuable question is which data can be read locally, which must be read from one place, and what consistency you are willing to give up to make the first category bigger — which is where this hands over to Agreement Costs Round Trips and to the consistency material in Database Engineering.
What actually helps
Reads are the tractable half. A regional read replica, an edge cache, or a CDN in front of anything cacheable removes crossings entirely for the majority of traffic, and the cost is staleness you can usually bound and reason about. This is the highest-leverage change available and it is mostly an operational one.
Writes are the hard half, because a write that must be globally consistent has to reach the authority, and the authority is somewhere specific. The options are all trades: accept the crossing on writes and keep reads local; partition data by region so each user's authority is near them; or relax consistency and reconcile — each with a different failure mode and a different amount of application complexity.
Two smaller wins are worth taking regardless. Terminating TLS at an edge close to the user removes handshake round trips from the long path, which matters disproportionately because a handshake is several crossings rather than one. And batching or collapsing chatty cross-region calls converts N crossings into one, which is the same boundary-counting logic as What Changes When Work Crosses a Machine applied to the most expensive boundary you have.
- Read replicas per region — removes crossings for reads; costs replication lag, which becomes a correctness question (see Replication Lag: Reads That Are Correct and Stale).
- Edge caching / CDN — removes crossings entirely for cacheable responses; costs staleness and invalidation complexity.
- Edge TLS termination — removes handshake round trips from the long path; costs edge infrastructure and certificate distribution.
- Regional data partitioning — each user's authoritative data is near them; costs cross-region queries becoming genuinely hard.
- Batching chatty calls — turns N crossings into one; costs larger payloads and coarser failure granularity.
- Moving compute to the data — often better than moving data to the compute, and usually cheaper than either alternative.
Key points
- Propagation delay is a physical floor: roughly 2 × distance ÷ 200,000 km/s, and no tuning gets under it.
- Real routes exceed great-circle distance and add switching delay, so observed RTT is always worse than the theoretical floor.
- The costly mistake is not multi-region deployment but incoherent topology — requests crossing an ocean several times per operation.
- Deploying stateless services regionally buys nothing if they still reach across the ocean for data; locality of *data* is what matters.
- Reads are tractable with replicas and caches; writes force an explicit consistency trade.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1EU user → US gateway: 90ms on the first crossing, before any application code runs.
- 2US API → EU database: the read crosses back to the EU where data residency requires the data to live — 90ms more.
- 3US API → US payment provider: 15ms, the one hop that happens to stay in a single region.
- 4US API → EU database: the write crosses again, 90ms, and cannot be avoided without changing where authority lives.
- 5US API → EU user: the response crosses a fourth time; ~360ms of the ~385ms total was propagation, and every service reported healthy latency.
- • "The services are slow in Europe." Every service is equally fast; the request is simply crossing an ocean repeatedly.
- • "We need bigger instances in the EU region." Propagation delay is unaffected by instance size.
- • "We deployed the service multi-region, so latency is fixed." Not if it still reaches across the ocean for data on every request.
- • "The network team should optimise the route." Route optimisation can recover the gap between observed and floor; it cannot beat the floor.
- • "Add a cache." Only helps for cacheable reads; the write path still crosses, and often that is what users are complaining about.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Latency segmented by client region, which makes a bimodal geography problem immediately obvious.
- • A distributed trace annotated with the region each span executed in — the crossing count is the diagnostic number.
- • Measured RTT between each pair of regions in your topology, compared against the distance-derived floor to see how much is route inefficiency.
- • Cacheable-response ratio per endpoint, which bounds how much of the traffic can avoid crossings entirely.
- • Read-versus-write ratio per endpoint, since the two have completely different mitigations.
- • Draw the actual request path with a region annotation per hop and count the crossings — this alone usually identifies the fix.
- • Move reads local with regional replicas or edge caches, accepting and bounding the resulting staleness.
- • Collapse chatty cross-region call patterns into batched or aggregated calls so N crossings become one.
- • Terminate TLS at an edge near the user, removing handshake round trips from the long path.
- • For writes, choose deliberately between accepting the crossing, partitioning data by region, or relaxing consistency — and write the decision down with its failure mode.
- • p50 and p99 latency for the affected client region specifically, compared against the same window before the change.
- • Crossing count per request from region-annotated traces, which should have measurably fallen.
- • Cache or replica hit ratio for the newly-local reads, confirming traffic actually avoided the crossing.
- • Replication lag and any correctness signal for reads now served locally, so a latency win is not silently a consistency regression.
- • Read replicas introduce lag, which turns some correctness questions into application-level problems.
- • Edge caching trades staleness for latency and adds invalidation as a permanent operational concern.
- • Regional data partitioning makes cross-region queries and global reporting genuinely difficult.
- • Multi-region infrastructure multiplies cost, deployment complexity and the number of ways the system can be partially down.
- • An alert on latency segmented by client region, so a topology change that reintroduces a crossing is visible.
- • A trace-derived check on cross-region crossing count for critical paths, failing when it increases.
- • A deployment review rule: any new dependency on a critical path states which region it lives in.
- • Periodic verification that regional replicas are actually serving regional traffic rather than silently failing over to a distant primary.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDRTT floors are derived as 2 × great-circle distance ÷ 200,000 km/s (approximately two-thirds of c, the propagation speed in optical fibre). This is a lower bound only.
- ENVIRONMENT-SPECIFICTypically-observed values depend on the specific cable routes, peering arrangements and provider backbone between two regions, and vary between providers and over time. Measure your own paths rather than relying on these figures.
- ILLUSTRATIVEThe four-crossing checkout budget is a constructed example of the ping-pong shape, not a measurement of a real system.