One Deploy, Five Services Down
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| User service team | A backwards-compatible change: rename name to full_name, keeping both fields for one release. | Nothing downstream can break — the old field is still present. | ✕ wrong |
| Billing service | Deserialisation failure: unknown field full_name. | A malformed response from the user service. | ✕ wrong |
| Canary | The user service’s own error rate and latency, both normal. | The change is safe to promote. | ✕ wrong |
| Platform team | Five services failing within two minutes of one deploy. | A platform-wide incident — networking or service mesh. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Adding a field is the canonical safe change and the old field was retained. Commit before reading on: name the property of the consumers that turned an additive change into a breaking one, and say what the canary would have needed to measure to catch it.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.