← IncidentsIntermediate

Every Dependency Is Fast and the Page Is Slow

Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.

The symptom

What was reported, before anyone knew what was happening.

The product page p99 is 1.9 seconds. It calls 30 backend services in parallel; each reports a p99 of 60ms and a p50 of 8ms. Three teams have each verified their own service is well within its SLO. The page has been slow for a year.

What each party saw, and what each concluded

The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.

WhoWhat they could actually seeWhat they concludedVerdict
Each backend service teamTheir own p99 at 60ms, comfortably inside a 100ms SLO.We are not the problem.✓ right
Page assembly serviceIts own processing at 4ms; the rest is waiting for the slowest of 30 responses.We are not the problem either.✓ right
Performance engineerNo single service ever appearing as the slow one twice in a row.It must be the network or a noisy neighbour.✕ wrong
Product ownerThirty green dashboards and one red page.Someone is lying about their metrics.✕ wrong

2 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.

Commit before you read on

Every service is inside its SLO and every claim is true. Commit before reading on: compute roughly what fraction of page loads must wait for at least one service’s p99, and state which number the SLOs should have been written against.

Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.

Everything below stays hidden until you commit. Being wrong here is the exercise — the answer looks obvious the moment you read it, which is exactly why reading it first teaches nothing.