Nineteen Days of Refunds in a Queue Nobody Watched
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
issued in the orders database with no corresponding payment-provider refund. The oldest was 19 days old. Every service involved reported healthy for the entire period.What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Orders service | Published a refund.requested message; the broker acknowledged it. | The refund is now someone else’s responsibility. Mark it issued. | ✕ wrong |
| Refund worker | A message whose amount field was a string, not a number. Threw on parse. Retried 3 times. Moved to the dead-letter queue. | Handled correctly — the message is unprocessable and I have preserved it. | ✓ right |
| Monitoring | Main queue depth 0, consumer error rate 0%, worker success rate 100%. | The refund pipeline is healthy. | ✕ wrong |
| Customer support | Occasional "where is my refund" tickets, resolved individually by manual refund. | Isolated bank delays. | ✕ wrong |
3 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Every component did exactly what it was designed to do, including the one that failed. Before reading on: say precisely which metric was missing, and why the three that existed all showed green.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.