Reading the Waterfall
Six shapes cover most of what a waterfall can tell you: the staircase, the comb, the fat leaf, the gap, the overhang and the cliff. Learning to recognize them turns trace reading from scrolling into diagnosis.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The gap is a finding, not empty space
The single most valuable habit in trace reading is noticing time where *no span exists*. A parent that starts at 0 and whose first child starts at 48 ms spent 48 ms doing something nobody recorded. That is almost never idle CPU: it is a thread-pool wait, a connection-pool acquisition, a GC pause, a cold-start initialization, or a middleware chain nobody instrumented. All of those are real user latency and none of them will ever appear in a "top spans by duration" list.
Gaps between siblings mean the same thing at a smaller scale — the parent did local work, or waited for something untraced, between two calls. Gaps at the *end* of a parent, after its last child returns, are usually response serialization or the framework writing the response, and they get large when payloads get large (JavaScript Costs Four Times, Not Once has the browser-side mirror of this).
The waterfall below has one deliberate 48 ms gap at the start. In a real investigation that gap is the entire finding: the service was not slow, it was queued, and the fix is concurrency limits or pool size, not code (Concurrency Limits: An Unbounded Server Is a Slower Server).
Six shapes and what each one means
Shapes generalize across stacks, which is what makes them worth memorizing. The staircase — children stacked end to end with no overlap — is serialized work that may not need to be; it is the most common recoverable latency in a trace. The comb — many narrow identical spans in a row — is an N+1 (The Comb: N+1 as a Visible Shape). The fat leaf — one wide span with no children — is either genuine local compute (profile it) or an uninstrumented call (instrument it), and telling those apart is what When the Trace Runs Out of Answers is for.
The gap is untraced time, discussed above. The overhang — a child extending past its parent's end — is a broken tree and means the durations cannot be trusted (Parents, Children and Links). The cliff — every span normal but the root much longer than their sum — is the same family: time is being spent outside anything you record.
Read shapes before durations. A 400 ms span in a trace whose baseline is 380 ms is noise; a comb of 100 × 2 ms spans in a trace that used to have three spans is a regression that no single duration will flag. The eye is very good at spotting repetition and empty space, and very bad at ranking numbers, so let it do the thing it is good at first.
| Shape | What you see | Most likely cause | Next move |
|---|---|---|---|
| Staircase | Children stacked end to end | Independent calls awaited one at a time | Can they run concurrently? (Sequential or Parallel: Same Work, Different Latency) |
| Comb | Many narrow identical spans | N+1 query or per-item network call | Batch or join (The Comb: N+1 as a Visible Shape) |
| Fat leaf | One wide span, no children | Local compute, or an uninstrumented call | Profile the process (Self Time, Total Time, and Where the CPU Went) |
| Gap | Dead time before or between children | Pool/thread wait, GC pause, cold start | Check saturation and pool metrics (Saturation: The Reading Utilization Cannot Give You) |
| Overhang | Child ends after its parent | Broken parenting or async nesting | Fix the tree before trusting any number |
| Cliff | Root ≫ sum of children | Untraced middleware or framework work | Instrument the entry path |
A long span is not the same as slow code
The distinction that separates useful trace reading from confident guessing: a span measures elapsed time, which is work plus waiting. A 120 ms inventory bulk lookup span in the caller might be 118 ms of the callee computing, or 3 ms of computing and 115 ms sitting in the callee's request queue, or 40 ms of TLS handshake because the connection pool was empty. All three render identically in the caller's waterfall.
Separating them requires the callee's own SERVER span. The difference between the caller's CLIENT duration and the callee's SERVER duration is network plus queueing — and when that difference is 90 ms you have found a saturated service, not a slow one. This is the single most useful arithmetic in distributed tracing, and it only works when both spans exist and are properly linked (Carrying the Trace Across the Gap).
The same logic applies inside a process. A wide span containing a database call is not evidence the database is slow until you compare it against the database's own reported execution time; the gap between them is client-side queueing, connection acquisition or result deserialization, which is exactly the diagnosis in Connection Pool Saturation: Waiting in Front of an Idle Database.
1caller: CLIENT inventory bulk lookup 120 ms2=> "inventory is slow, ask that team to optimize their query"3 4# Nothing here supports that conclusion. The 120 ms includes:5# connection acquisition, network, the callee's own queue wait,6# the callee's work, serialization, and network back.1caller: CLIENT inventory bulk lookup 120 ms2 |- (gap 0-> 38 ms) 38 ms connection pool wait3callee: SERVER GET /inventory/bulk 74 ms4 |- CLIENT postgres SELECT 12 ms5 |- (self time) 62 ms <- actual work6network + queueing = 120 - 38 - 74 = 8 ms7 8=> the finding is a 38 ms pool wait in the CALLER,9 plus 62 ms of callee compute worth profiling.10 The query is not the problem.The first reading blames another team on the strength of a number that contains at least five different things. The second locates 38 ms in your own connection pool — a fix you own and can ship today.
Key points
- Read geometry before durations: gaps and repetition are findings the eye catches instantly and a sorted duration list hides completely.
- A gap before the first child is untraced time — pool wait, GC pause, cold start, uninstrumented middleware — and it is real user latency.
- Six shapes cover most traces: staircase (serialization), comb (N+1), fat leaf (profile it), gap (saturation), overhang and cliff (broken or missing instrumentation).
- A span measures elapsed time, which is work plus waiting; only comparing the caller's CLIENT span with the callee's SERVER span separates the two.
- Overlapping siblings mean concurrency and stacked siblings mean serialization — the fastest way to spot recoverable latency.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Load → thread pool: request arrives while all pool threads are busy, so it waits 48 ms before any handler code runs.
- 2Handler → trace: instrumentation starts at the handler, so the 48 ms is outside every span and appears as a gap.
- 3Waterfall → engineer: the visible spans sum to 272 ms while the root reads 320 ms, and nothing in the "slowest spans" view explains it.
- 4Pool metrics → engineer: queued-request count and pool utilization confirm saturation, which the trace could only hint at.
- • "There is no span there, so nothing was happening." Untraced is not idle. Gaps are the highest-value finding in a waterfall.
- • "The widest bar is the bottleneck." Only if it is on the critical path and only if it changed. A wide parallel span that finishes early costs nothing (The Critical Path Is the Only Path That Pays).
- • "The callee is slow." Not until you have subtracted pool wait and network from the caller's number.
- • "The trace looks the same as last week, so nothing regressed." Compare span *counts* too: same shape with 100 extra narrow spans is a serious regression.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • For the slow trace, compute parent duration minus the span of its children; anything above ~10% of the request is worth instrumenting.
- • Diff the span count and shape against a fast trace for the same endpoint before comparing durations.
- • For each cross-service call, subtract the callee's SERVER duration from the caller's CLIENT duration to isolate network plus queue wait.
- • Check whether concurrent-looking calls actually overlap on the timeline rather than assuming the code's `Promise.all` did what you think.
- • Instrument the gap first — middleware, pool acquisition, deserialization — because you cannot optimize time you cannot attribute.
- • Collapse staircases into concurrent calls where the work is genuinely independent.
- • Raise pool or concurrency limits only after confirming the wait is queueing and the downstream can absorb it ([[concurrency-limits]]).
- • Profile fat leaves rather than guessing at them; a wide childless span is a question, not an answer.
- • The gap should shrink or become an explicit span after instrumentation — either outcome is progress, since the second one localizes it.
- • After parallelizing, the waterfall should show overlap and the root duration should fall by roughly the sum of the removed serial segments.
- • Endpoint p99 should move in the same direction and magnitude as the trace suggests; if it does not, the trace was not representative.
- • Re-check that the critical path moved to a different span rather than the fix simply shaving the same one ([[bottleneck-migration]]).
- • Instrumenting gaps adds spans, which adds cost and can itself add measurable overhead on very hot paths.
- • Parallelizing raises peak concurrency against downstreams — you can convert your latency problem into their saturation problem.
- • Larger pools trade queueing for memory and downstream pressure, and past a point make things worse.
- • Reference waterfalls go stale and need maintaining, or they cause false confidence during the next incident.
- • Track spans-per-trace as a metric per endpoint; it catches N+1 regressions that latency alone can miss under low load.
- • Alert on root-span duration minus summed child duration exceeding a threshold — a generic "we lost visibility" detector.
- • Keep a reference waterfall for critical endpoints in the runbook so the next responder has a shape to diff against.
- • Add a load test that asserts pool wait stays near zero at expected concurrency (Load Testing: What Question Is This Test Answering?).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 320 ms dashboard trace is constructed to contain one clean example of each shape worth recognizing.
- ENVIRONMENT-SPECIFICWhether pool acquisition, middleware and serialization appear as spans depends entirely on your framework and which auto-instrumentation is enabled.