Where the Request Actually Went
Metrics tell you the endpoint got slower. A trace tells you which of the eleven things it touched got slower. One request, one timeline, every hop measured — and usually one span holding 80% of the budget that nobody suspected.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The question metrics structurally cannot answer
Metrics are aggregates. http_request_duration_seconds for the checkout endpoint tells you the p99 doubled; it cannot tell you that the doubling lives entirely in a payment provider call, because by the time the number reached your dashboard the individual request had been summed into a histogram bucket alongside every other request. The per-request path — the thing you actually need — was discarded at collection time. That is not a gap in your dashboard; it is what an aggregate is.
A trace keeps the path. One request gets one trace id, every unit of work inside it gets a span carrying that id plus its own start time, duration and parent, and the backend reassembles the tree. Reading it as a timeline turns "the endpoint is slow" into "the payment provider call is 410 of the 489 ms", which is a different sentence: the first one starts an argument between teams, the second one ends it.
The worked example below is the shape you will see most often. Four services and a cache all behave acceptably. One external dependency holds 84% of the wall clock. Every internal dashboard is green, which is exactly why the on-call engineer spent forty minutes looking at the database before opening a trace.
A trace is a tree that happens to be drawn as a timeline
The mental model that survives contact with real traces: a trace is a tree of causally related work, and the waterfall is just one rendering of it. The tree structure is what lets you ask "what did the API do?" and get an answer bounded to that subtree. The timeline rendering is what lets you ask "what overlapped with what?" — a question the tree alone cannot answer, because a tree has no notion of wall clock.
Both readings matter and they catch different bugs. The tree reading catches missing instrumentation: if the checkout-api span has 474 ms of duration and its children account for 454 ms, the remaining 20 ms is the API's own work — fine here, but if that gap were 300 ms you would have found an uninstrumented call. The timeline reading catches false serialization: three calls that could have run concurrently but are stacked end to end, which is Sequential or Parallel: Same Work, Different Latency.
Note what the trace does not tell you. It says the payment call took 410 ms; it does not say whether that was the provider computing, the network, TLS setup, or your own client library queueing behind a saturated connection pool. Answering that requires either instrumenting inside the client (a child span for connection acquisition) or a different signal entirely. Traces bottom out at the edge of your instrumentation, and knowing where that edge is prevents a lot of confident wrong conclusions.
What tracing buys, and the bill it sends
Tracing is the most expensive of the four signals per unit of insight, and the most valuable when the question is "where". Every span is a structured record shipped over the network and stored; a request touching thirty spans across eight services produces thirty records for one user action. At meaningful traffic this is why Sampling Without Throwing Away the Evidence is not optional and why "just trace everything" is a budget decision disguised as an engineering one.
The other cost is instrumentation debt. Traces are only as good as the least-instrumented hop: one service that does not propagate context turns a single 489 ms trace into two disconnected traces that no query will ever join, and the gap looks like the caller was idle. This is the failure mode covered in Carrying the Trace Across the Gap, and it is the single most common reason a tracing rollout disappoints.
Against that, tracing is the only signal that answers "where" without a hypothesis. Metrics require you to already suspect a component to go look at its dashboard; profiles require you to already suspect a process. A trace hands you the ranked list of suspects for free, which is why it is usually the second thing to open after the metric that told you something is wrong at all.
| Question | Signal that answers it | Signal that will mislead you |
|---|---|---|
| Is anything wrong at all? | Metrics: endpoint p99 against its baseline | Traces — you cannot eyeball a million of them |
| Where in the request path did the time go? | Traces: the span tree with durations | Metrics — per-service averages hide a slow subset |
| Why is *this* service slow, given no slow children? | Profiles: CPU or allocation inside the process | Traces — a leaf span has nothing left to decompose |
| What exactly happened to this one failed request? | Logs, joined by the trace id | Metrics — a counter cannot tell you which user |
Key points
- Metrics tell you an endpoint is slow; only a trace tells you which hop inside it consumed the budget, because aggregation discards the per-request path.
- A trace is a tree rendered as a timeline: the tree bounds "what did this service do", the timeline reveals what overlapped and what needlessly serialized.
- A parent span longer than the sum of its children means either real local work or a missing instrumentation — check which before theorizing.
- Traces bottom out at the edge of instrumentation: "the provider took 410 ms" may be their compute, the network, or your own pool wait.
- The cost is real — spans per request multiply by traffic, which forces sampling and makes propagation gaps expensive.
Progressive depth
Overview
One request gets one id. Every piece of work it causes records how long it took and who called it. Reassembled, that is a picture of where the time went — the thing no dashboard can show you, because dashboards average requests together.
Practical
Open a slow trace, sort spans by duration, compute self time (duration minus children). Rank by change from baseline rather than absolute size. The top changed span is your suspect; the parent-minus-children gap is your uninstrumented blind spot.
Advanced
Read the timeline as well as the tree. Stacked independent calls are recoverable latency (Sequential or Parallel: Same Work, Different Latency); a gap before the first child is queueing or pool acquisition, not idleness; a repeated narrow span is an N+1 (The Comb: N+1 as a Visible Shape). Only spans on the critical path repay optimization (The Critical Path Is the Only Path That Pays).
Internals
A span is a structured record: trace id, span id, parent id, start, duration, attributes, status. Context travels in-band (a traceparent header) while spans travel out-of-band to a collector, which is why a trace can be assembled at all — and why a hop that drops the header breaks it irreparably. See Trace, Span, Attribute, Status and Carrying the Trace Across the Gap.
Trace Waterfall & Critical Path
Change an input and watch which number moves — and which one does not.
Of 840 ms, 620 ms sits in one third-party call. Making user.profile instant would save zero milliseconds, because it finishes while billing.summary is still running. That is the entire argument for reading the critical path before choosing what to optimize.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Users → support: "checkout is slow", with no pattern by region or account.
- 2Endpoint metrics → on-call: p99 for
POST /checkoutmoved 140 ms → 500 ms; error rate flat, traffic flat, so this is not load. - 3Per-service dashboards → on-call: every service reports normal p99, because none of them owns the request end to end.
- 4Trace of one slow request → on-call:
payment-provider POST /chargesholds 410 of 489 ms; every other span is at its usual duration. - 5Provider status page → team: elevated latency in one processing region, confirming the span rather than the other way round.
- • "Every service dashboard is green, so the problem must be the network." Green per-service dashboards are exactly what a slow third party produces, because the third party is on nobody's dashboard.
- • "The gateway span is 489 ms, so the gateway is slow." A parent span includes everything it waited for; the gateway's own contribution here is 10 ms.
- • "One trace proved it." One trace is a hypothesis. Confirm against a set of slow traces and against the pre-regression baseline before paging a vendor.
- • "The database span is 40 ms, that seems high, start there." It is the second-largest span and it is unchanged from baseline. Rank by *delta from normal*, not by absolute size.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Pull traces for the slow endpoint filtered to `duration > p95` over the regression window, not a random sample — the fast ones do not contain the bug.
- • Sort the spans in one slow trace by duration and read the top three; compare against a trace from before the regression, same endpoint.
- • Compute self time per span (span duration minus the sum of its children) to separate "this service is slow" from "this service waited on someone".
- • Check span count per trace as its own number: a jump from 12 to 112 spans is an N+1, not a latency problem ([[n-plus-one-traces]]).
- • Fix the span that actually holds the budget — here, take the payment provider off the synchronous path: authorize asynchronously and confirm via webhook, so user-visible latency stops depending on a third party ([[llm-latency]] has the same shape for model calls).
- • If the slow hop must stay synchronous, bound it: an aggressive timeout plus a documented fallback converts an unbounded tail into a known, fast failure ([[timeouts-and-latency]]).
- • Where the trace shows serialized independent calls, parallelize them — this is free latency that costs only concurrency ([[sequential-vs-parallel]]).
- • Where the parent-minus-children gap is large, instrument the gap before optimizing anything: you cannot fix time you cannot see.
- • Re-pull slow traces after the change: the payment span should be off the critical path entirely, or bounded by the timeout you set.
- • Compare endpoint p99 over a full traffic cycle (same hours, same days) against the pre-change baseline — not against the incident window, which flatters any fix.
- • Confirm the critical path moved rather than the total shrinking by luck: the new top span should be a different one ([[bottleneck-migration]]).
- • Watch the error and fallback rates alongside latency; a timeout that "fixed" p99 by failing 3% of checkouts is not a fix.
- • Tracing costs storage and network per span, and the bill scales with traffic times span count — the reason sampling exists.
- • Instrumentation is ongoing work: every new service, client library and queue hop is another chance to break the trace.
- • Moving a dependency off the synchronous path buys latency and costs consistency — the user now gets an answer before the work is truly done.
- • Aggressive timeouts convert slow success into fast failure, which is better for latency dashboards and worse for anyone whose payment was cancelled at 401 ms.
- • Alert on the endpoint SLI, not on the provider — you care when users are slow, whoever caused it (Alerts Worth Waking Someone For).
- • Keep a saved trace query for
POST /checkout duration > 1sso the next investigation starts with evidence instead of a hunt. - • Add a dedicated histogram for external-dependency duration by provider so the next slow third party shows up in metrics, not only in traces.
- • Tail-sample so slow and errored traces are always retained; a 1% uniform sample would have kept none of the traces that mattered here.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 489 ms breakdown is constructed to show the common shape — one external dependency dominating an otherwise healthy request. Real numbers depend entirely on your dependencies and traffic.
- ENVIRONMENT-SPECIFICWhether a "410 ms provider call" is the provider, the network, TLS setup or your own connection-pool wait depends on where your client library is instrumented.