The Revenue Report That Was Wrong Only on Mondays
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Stream processor | Events arriving with event timestamps older than the current watermark. | These are late events past the allowed lateness; drop them, as configured. | ✓ right |
| Mobile client | A device offline over the weekend, buffering purchase events, flushing them on reconnect Monday morning. | Delivering my events, each stamped with the time the purchase actually occurred. | ✓ right |
| Analytics consumer | A completed hourly window emitted on time with a total. | This window is final and complete. | ✕ wrong |
| Data engineer | Zero errors, zero lag, zero DLQ entries. | The pipeline is fine; the discrepancy is a query or timezone bug in the report. | ✕ wrong |
2 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
The processor is doing exactly what it was configured to do and reports no errors. Commit before reading on: name the two different notions of time in play, and say which of them the report’s consumers believed they were reading.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.