Distributed Tracing
Where a request spends its time across services. Spans and their relationships, context propagation through queues, the waterfall view, critical-path reasoning, and N+1 as a visible shape.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
Metrics tell you the endpoint got slower. A trace tells you which of the eleven things it touched got slower. One request, one timeline, every hop measured — and usually one span holding 80% of the budget that nobody suspected.
A span is a timed operation with a parent, a status and a bag of attributes. Which facts belong in attributes, which belong in span events, and which belong in a metric instead is the difference between a trace you can query and a very expensive log line.
Nesting is a claim about causality and containment: a child span asserts its parent was waiting for it. Get that wrong — most often by making a queued job a child of the request that enqueued it — and the waterfall stops describing anything real.
Trace context travels in-band with the work: a header on the HTTP call, a field on the queue message, an argument to the job. Every hop that forgets to carry it cuts the trace in half — and the caller looks like it was idle for 400 ms.
Six shapes cover most of what a waterfall can tell you: the staircase, the comb, the fat leaf, the gap, the overhang and the cliff. Learning to recognize them turns trace reading from scrolling into diagnosis.
In a fan-out, only the slowest branch controls when the request finishes. Optimizing any other branch produces a beautiful graph in your dependency dashboard and zero improvement for users — until the critical path moves, and then a different branch matters.
One query to fetch the users, then one query per user to fetch their orders. Every individual query is fast, every dashboard is green, and the endpoint takes 268 ms because it made 101 round trips instead of 2.
At 10,000 requests a second, tracing everything is a second production system. Sampling is inevitable; the question is whether you keep a random 1% — which discards almost every slow and failed request — or keep the ones that matter.