Reading a Flame Graph
Width is time, height is stack depth, and the horizontal axis is not time at all. Getting that last part wrong is the single most common flame-graph misreading, and it makes people look for patterns that cannot exist.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The three rules
A flame graph is an aggregation of stack samples, drawn with three conventions. Width is proportional to the samples containing that frame — wider means more time. Height is call depth: a frame sits on top of its caller. And the horizontal axis is ordering only, not time — frames are laid out alphabetically (or by some stable order) so that identical stacks merge into one wide block. Nothing in a flame graph runs left to right.
That third rule is where most misreadings begin. People look for phases ("first it parses, then it computes"), read a left-hand frame as "early", or try to spot a slow period. None of that is in the picture. A time-ordered rendering exists and is a different tool: a flame *chart* keeps the x-axis as time and does not merge stacks. If you need to see phases or a specific slow request, you want a flame chart or a trace, not a flame graph.
What the merging buys is the ability to see aggregate cost across thousands of samples at a glance. A function called from twelve places appears as twelve narrow frames in a flame chart and, in a flame graph, as one wide frame per unique stack — which is why a flame graph answers "where does the CPU go" far better than any list.
[sortByRelevance 38%]
[normalizeWeights 14%][.................]
[calculateScore .......................................55%]
[parseReq 5%][auth 10%][json.serialize 15%][calculateScore ...............55%]
[handlers.search ..................................................... 71%]
[router.handle ....................................................... 98%]
[server.listen ....................................................... 99%]
[main ................................................................100%]
read the PLATEAUS, not the towers:
sortByRelevance is a wide top -> 38% self time, the target
main is the widest frame -> 0% self time, pure scaffoldingPlateaus, towers and what each means
The shapes worth naming. A wide plateau at the top is a function doing a lot of work itself — the highest-value target, because that is self time with nothing beneath it to blame. A tall narrow tower is deep call nesting that costs little; recursion produces impressive-looking spires that are frequently irrelevant. A wide base narrowing sharply means work spread across many small callees, so no single fix helps much and the caller's structure is the question.
Two frames worth learning to spot: runtime frames (GC, allocation, JIT) appearing as wide plateaus mean the problem is memory or compilation rather than your logic (Garbage Collection: Pause, Throughput, Footprint — Pick Two, JIT and Warm-Up: The First Thousand Requests Are a Different Program); and the same function appearing in several distant places means it is called from multiple paths, and the total cost is the sum of those widths, which no single frame displays. Most tools support "search and highlight" for exactly this — highlight a function name and the tool sums its width across the graph.
The differential flame graph is the version to reach for during a regression. Rendered from two profiles, it colours frames by change — grown frames one colour, shrunk another — which turns "read two mountain ranges and spot the difference" into a picture that points at the answer. When you have a good and a bad window, this is almost always the right rendering (When the Trace Runs Out of Answers).
| Shape | Meaning | Next move |
|---|---|---|
| Wide plateau at the top | High self time in one function | The primary target — read the code |
| Tall narrow tower | Deep nesting, little cost | Usually ignore; recursion is not automatically a problem |
| Wide base, narrowing fast | Cost spread over many small callees | Look at the caller's structure, not any single callee |
| Wide runtime frames (GC, alloc) | Memory or compilation cost, not your logic | Switch to allocation profiling (Allocation Rate Is a Cost Even Without a Leak) |
| Same frame in several places | Called from multiple paths | Use search-and-highlight to sum its true share |
| Nearly flat, very short graph | Little CPU work is happening at all | Wrong tool: the process is waiting (Computing or Waiting?) |
The misreadings, in order of frequency
The x-axis misreading is first and worst: treating horizontal position as time produces confident nonsense about execution order. Second is reading the widest frame as the target, which is always main or the framework entry — the useful frames are the widest ones *with little above them*. Third is ignoring sample count: a flame graph from 40 samples is a picture of noise rendered at full confidence, and nothing in the visualization tells you that.
Fourth, more subtle: flame graphs of wall-clock or off-CPU profiles read differently. An off-CPU flame graph shows where threads *blocked*, so a wide epoll_wait plateau is entirely normal and means the service was waiting for work — not a bug. Applying CPU-profile intuitions to an off-CPU graph produces the classic "our biggest hot spot is idle" confusion.
Finally, inlining and missing symbols. Aggressive inlining merges callee cost into the caller, so the frame you want may not exist as its own block; missing symbols render as addresses. Both are fixable with build flags, and neither is visible unless you check — a flame graph of hex addresses is a sign to fix the toolchain before continuing.
1"parseRequest is on the left, so it runs first,2 then auth, then serialize, then calculateScore.3 The slowdown starts about two-thirds of the way across."4 5# All four statements are meaningless.6# The x-axis is alphabetical ordering of merged stacks.7# There is no 'across'. There is no 'starts'.1"sortByRelevance is a wide plateau at the top: 38% of2 samples were executing inside it, aggregated over the3 whole 30 s window.4 5 calculateScore is 55% total but only 17% self, so most6 of its cost is in its two children.7 8 3,102 samples, so 38% vs 36% would be noise but 38% vs9 9% in the baseline profile is a real change."The second reading makes only claims the visualization supports: aggregate share, self versus total, and a difference large enough to exceed sampling noise. It also names the sample count, which is what makes the percentage comparison legitimate.
Key points
- Width is share of samples, height is stack depth, and the horizontal axis is ordering — never time. Flame charts keep time; flame graphs do not.
- Read the wide plateaus near the top: those are self time. The widest frame overall is always the entry point and always useless.
- Tall narrow towers are deep nesting that usually costs little; impressive spires are not findings.
- Wide runtime frames (GC, allocation, JIT) redirect the investigation from your logic to memory or compilation.
- Differential flame graphs — good window versus bad window — turn regression hunting into reading colour rather than comparing two mountain ranges.
Progressive depth
Overview
Each block is a function. The wider it is, the more of the CPU it used. Blocks sit on top of whoever called them. Look for wide blocks near the top — that is where the time is going.
Practical
Rank by wide plateaus with little above them (self time), not by the widest frame (always the entry point). Check the sample count before believing a percentage, and use search-and-highlight to sum a function called from several places.
Advanced
Use differential flame graphs between a healthy and an unhealthy window — reading colour is far more reliable than comparing two shapes. Watch for runtime frames (GC, JIT) as a redirect to Allocation Rate Is a Cost Even Without a Leak, and remember off-CPU graphs invert the meaning of a wide plateau.
Internals
The graph is built by collecting stack samples, folding identical stacks into counted strings, and laying them out with a stable horizontal ordering so identical prefixes merge into one block. Width is therefore sample count, which is a statistical estimate of time — never an exact measurement. Inlining merges callee frames into callers before the profiler ever sees them, so some functions cannot appear at all.
Flame Graph Reader
Change an input and watch which number moves — and which one does not.
Left-to-right ordering carries no meaning — it is not time. A flame graph shows where samples landed, not the sequence they landed in. Reading it as a timeline is the most frequent mistake, and it leads people to "optimize the thing on the left first".
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Deploy → code path: a scoring change adds a sort inside a per-item loop.
- 2Loop → CPU: sample stacks increasingly land inside
sortByRelevance, raising its share from 9% to 38%. - 3Flame graph → engineer: a wide plateau appears at the top of the
calculateScoresubtree that was not in last week's profile. - 4Differential graph → engineer: the frame is coloured as grown, and its callers identify the loop that gained the sort.
- • "The graph shows the request going left to right." It does not. Horizontal position carries no temporal meaning whatsoever.
- • "
mainis the widest frame, so the problem is in startup."maincontains everything by construction and has essentially no self time. - • "This deep recursion tower is the problem." Height is depth, not cost. A ten-level tower two pixels wide is 0.2% of your CPU.
- • "
epoll_waitis our biggest hot spot." On an off-CPU graph that means the service was idle waiting for work, which is what a healthy server does.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Check sample count before interpreting anything; a few hundred samples minimum before percentage differences mean much.
- • Use search-and-highlight to sum a function's width across all the paths that call it, rather than reading one block.
- • Compare self versus total for the wide frames — a wide frame with a wide child is not itself the cost.
- • Render a differential flame graph against a healthy baseline whenever you have one; it is strictly more informative than a single graph.
- • Target the widest plateau with little above it, after confirming its share changed relative to baseline.
- • Fix the caller when the cost is spread thinly across many small callees — no single frame will repay optimization.
- • Redirect to allocation or GC work when runtime frames dominate, rather than optimizing application code that is not the cost.
- • Fix symbolization and inlining flags before drawing conclusions from a graph full of addresses or suspiciously absent functions.
- • Re-render the flame graph after the change: the target plateau should shrink toward its baseline width.
- • Differential graph against the pre-fix profile should show the frame as shrunk and no new plateau appearing elsewhere.
- • CPU seconds per request should fall — the graph is evidence about mechanism, the metric is evidence about effect.
- • Confirm the user-facing latency metric moved; a narrower plateau on a non-critical path changes nothing users can feel.
- • Flame graphs aggregate away per-request detail, so they cannot answer "why was *that* request slow" — you need a trace or flame chart.
- • Differential graphs need a trustworthy baseline, which means retaining profiles and the storage that implies.
- • Full symbolization can require shipping debug symbols, which has size and, in some contexts, disclosure implications.
- • Reading flame graphs well is a learned skill; the visualization is unusually easy to misread with confidence.
- • Keep profiles per release so a differential graph across deploys is one command rather than an investigation (Always-On Profiling, and the Diff That Finds Regressions).
- • Alert on CPU seconds per request rather than on profile shape, which is not a thing you can alert on.
- • Store the flame graph from the incident in the review so the next responder has a known-bad shape to compare against.
- • Verify symbolization in CI for release builds, so production profiles are readable when you need them.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe ASCII flame graph is a constructed rendering of the same illustrative profile used in Self Time, Total Time, and Where the CPU Went; real graphs have many more small frames.
- RUNTIME-SPECIFICWhether GC, JIT and inlined frames appear at all — and under what names — depends on the runtime and the profiler's symbolization support.