← IncidentsBeginner

Charged Twice, and Nothing Errored

Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.

The symptom

What was reported, before anyone knew what was happening.

Forty-seven customers were charged twice for the same order on a Tuesday afternoon. Both charges appear on the card statement 2 to 3 seconds apart. The payments service reported a 0% error rate for the entire window, and the checkout service logged no exceptions. The only signal was support tickets.

What each party saw, and what each concluded

The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.

Who	What they could actually see	What they concluded	Verdict
Checkout service	The call to Payments produced no response within its 2s deadline; the connection returned `504 Gateway Timeout`.	The charge did not happen, so it is safe — and necessary — to retry.	✕ wrong
Payments service	Two `POST /charge` requests with different `Idempotency-Key` values, same card, same amount, 2.4 seconds apart. Both authorised successfully in 2.3s and 0.4s.	Two distinct payment intents from one customer. Nothing to deduplicate.	✕ wrong
The proxy between them	An upstream response that had not arrived after 2000ms.	The upstream is unhealthy; synthesise a 504 and close the connection.	✕ wrong
On-call engineer	Payments dashboard: error rate 0%, p50 340ms, p99 2.31s. Checkout dashboard: 47 retries, all successful.	Payments is healthy, so the defect is on the checkout side.	✓ right

3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.

Commit before you read on

Payments already has an idempotency layer, and it was running correctly the whole time. Before you read on, name the specific reason it did not collapse the second charge into the first.

Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.

Everything below stays hidden until you commit. Being wrong here is the exercise — the answer looks obvious the moment you read it, which is exactly why reading it first teaches nothing.