Two Regions, Two Versions of the Same Customer
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| EU region | Writes accepted locally; replication queue to US growing. | The US region is down. Continue serving; we will catch up. | ✕ wrong |
| US region | The mirror image: writes accepted locally, replication queue to EU growing. | The EU region is down. | ✕ wrong |
| Support agent in Dublin | A customer’s record with the address they had just corrected. | The correction is saved and globally visible. | ✕ wrong |
| Conflict resolver (last-write-wins on wall clock) | Two versions of the same record with timestamps 900ms apart. | The later timestamp is the newer edit; keep it. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Both regions stayed available and both were configured for active-active multi-leader writes, as intended. Commit before reading on: which of the two symptoms — the mixed records or the 400 lost edits — is a design consequence, and which is a bug?
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.