← IncidentsIntermediate

The Trace Where the Response Preceded the Request

Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.

The symptom

What was reported, before anyone knew what was happening.

While debugging a checkout failure, the merged log timeline shows the payment service’s "charge authorised" line 700ms *before* the order service’s "calling payments" line. Several traces show child spans starting before their parents. Two engineers spent a day building a theory around a rogue background job.

What each party saw, and what each concluded

The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.

WhoWhat they could actually seeWhat they concludedVerdict
Order service hostIts own clock reading 14:22:03.100 when it emitted "calling payments".This event happened at 14:22:03.100.✕ wrong
Payments hostIts own clock reading 14:22:02.400 when it emitted "charge authorised".This event happened at 14:22:02.400.✕ wrong
Log aggregatorTwo events with timestamps 700ms apart.Sort by timestamp; the earlier one happened first.✕ wrong
Debugging engineerAn effect ordered before its cause.A second, hidden caller is triggering payments — a rogue job.✕ wrong

4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.

Commit before you read on

Both hosts run NTP and both report themselves synchronised. Commit before reading on: what is the *only* ordering claim you can make about these two events without trusting either clock, and what would you have to have recorded to make it?

Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.

Everything below stays hidden until you commit. Being wrong here is the exercise — the answer looks obvious the moment you read it, which is exactly why reading it first teaches nothing.