Highest Useful Level First
Solve the problem at the highest level where it can be stated and checked, and go deeper only when that level cannot explain what you see. Most problems are solved at the top; the ones that are not announce themselves by a promise that holds and a symptom that persists.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
At which level should you try to solve this problem first — and what tells you that level was not enough?
"Search products by name is slow." I know the answer is probably an index. I also know that last time I "knew" it was an index I spent two days on the query planner and the real cause was the page fetching every product and filtering in JavaScript. I want a rule for where to start.
Start where the expertise is. The engineer who knows databases opens the query plan; the one who knows frontend opens the network tab; the one who read about SSDs last week thinks about disks. Each starts at the level they can do something at.
The level chosen by expertise is often below the problem. A query plan is examined for a query that runs fast; the slowness was a request that fetched everything, and no amount of index tuning changes how much is fetched.
- The level chosen by expertise is often below the problem. A query plan is examined for a query that runs fast; the slowness was a request that fetched everything, and no amount of index tuning changes how much is fetched.
- Starting low is expensive per hypothesis: an index takes a migration, a page-cache theory takes a benchmark, and each one that fails to help has consumed a day. Starting high is cheap per hypothesis: "is the slow part the request, the service or the query?" is answered by timing.
- Fixes made low hide problems that live high. An index that makes the fetch-everything query fast is a real improvement that leaves the design — fetch everything, filter in the browser — in place to fail again at the next order of magnitude.
- Because "I knew it was the index" was once right, it becomes the starting point every time, and the engineer stops generating hypotheses at the levels above, where the boring, common causes are.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Start at the highest level where the problem can be stated in its own words and checked with an observation. "Search is slow" is stated at the request level and checked by timing the request. If the top level can be checked, check it before descending; a problem explained at the top does not need the bottom.
- Descend only on a specific signal: the current level's promise *holds* and the symptom *persists*. The handler is fast but the request is slow — descend to the network. The service is fast but the handler is slow — the problem is between them. Each descent is justified by a contract that was checked and found intact.
- At each new level, restate the problem in that level's words before touching its tools. "Search is slow" becomes "this query reads many pages" only once timing has put the time inside the query; before that, the database vocabulary is a guess wearing a lab coat.
- Stop at the level where a fix removes the symptom and the fix is stated in that level's words. If the fix is an index, the problem was a database problem; if the fix is "do not fetch every product", it was a service problem, and the index would have been a bandage over it (Going One Layer Deeper).
The descent, as a loop
The pipeline is the move made mechanical: state at the top, check, split at the next contract, descend only where the symptom went. It ends when a fix can be stated in the current level's words. Each step's failure is the reflex in disguise.
- 1State at the highest level
the problem in the words of the level where it was reported, with one confirming observation
fails by stating it in the words of the level you already suspect
- 2Check the promise here
observe whether this level's promise is broken or holds
fails by checking with an observation from a different time, data set or request
- 3Split at the next contract
measure on both sides of the boundary below; the symptom is on one side
fails by descending without measuring, because "it is probably the database"
- 4Restate in the new level's words
"slow request" becomes "reads every page", or "holds a transaction across a network call"
fails by using the new level's tools before you can say the problem in its words
- 5Fix where the cause is stated
a fix in this level's vocabulary; record any lower-level bandage as a bandage
fails by a fix one level down that hides the promise broken here
Where to look first
The decision is the one the engineer in the situation wanted: a rule for where to start. It is not "always the top"; it is the highest level at which the problem is *stated*, and the options are the shapes that come up.
Where does the investigation start?
when the symptom is "slow", "fails sometimes" or "wrong response" as seen by a user or a client
cost a timing or a captured response; cheap, and it rules out half the stack immediately
when the symptom is stated in domain words — total, stock, status — and the response arrived fine
cost a log of the decisions the service made; cheap, but only if the service logs its inputs
when timing has placed the time inside one query, or the symptom is a constraint or a deadlock error in the database's own words
cost a plan, an index or a migration; expensive per hypothesis, so earn it first
when the symptom is stated there already — fsync latency, page faults, a device error — or every level above has been cleared
cost benchmarks and machine access; rarely the level, and never the first guess for an application symptom
Slow search, twice
The same feature, the same symptom, a month apart, with different causes at different levels. The compare is the point of the lesson: the correct starting level was identical both times, and the descent stopped at different floors because the signal did.
First time: index added to a query that was not the problem; the endpoint still returns every product. Second time: the index is the right fix, found by the same guess, which now looks vindicated — and the fetch-everything design was never noticed.
First time: the time is in the response size; fix at the request level — search on the server, return a page. Second time: the time is in one query; descend, restate as "no index on name", add the index. Two causes, two levels, one method.
Descending on measured signal finds the level where the promise broke each time; guessing the level finds the guess, and is right only when the guess is.
How to do it
Most important first.
- State the problem at the top level and name the observation that would confirm it there. For "slow": a timing at the request boundary.
- Split the timing at the next contract down: request vs handler, handler vs service, service vs query. Descend to whichever side holds the time; do not descend where the time is not.
- Before opening a level's tools, rewrite the problem in that level's words. If you cannot, you have not earned the descent yet.
- When a fix works, say at which level it was stated. A fix below the level of the cause is a bandage — write it down as one, so it is revisited (Measure Before You Optimize in the performance domain holds the measurement discipline).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- "Search products by name is slow." Top level: the request takes longer than the budget — confirmed by timing. Split: the handler returns in a fraction of that; the rest is response size and transfer. The problem is at the request level: the endpoint returns every product and the browser filters. Fix at that level: search on the server with a
nameparameter, return a page of results. The database was never opened; the "obvious" index was for a query that did not exist yet. - Same feature, a month later, after the fix: slow again, and now the timing puts the time inside the handler, inside the service, inside one query. The level has been earned: restate as "this query filters by name without an index and reads every page". Now the query plan is the right tool, and the index is the right fix — a database-level fix for a database-level problem (Should I Add an Index? in the database domain decides which).
- Checkout that is slow only under load. Top: request timing confirms. Split: handler fast in isolation, slow under load — the problem is not inside any level but between requests: a shared resource. Restated at the service level as "checkout holds a transaction open while it calls the payment provider", which is a service-level design problem, and the fix is moving the call outside the transaction, not a bigger database (External Systems Fail).
How you know it worked
What now exists that did not before, and what question you can now ask.
- Each descent in your notes is preceded by a contract that was checked and held; there are no descents "because it might be".
- The fix is stated in the words of the level where the cause was found, and you can say why that level and not one below it.
- The engineer who knows databases spent the first half hour timing the request, and only then, if at all, opened the planner.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What is the highest level at which this problem can be stated in its own words, and what single observation would confirm it there?
- ?Which contract between two levels do I split the observation at next, and which side holds the symptom?
- ?Has the level I am about to descend into been earned by a promise that held above it?
- ?Is my fix stated in the words of the level where the cause was, or one below it?
What can go wrong
- Never descending. The top level is checked, the fix is a cache in front of the slow endpoint, and the query that reads every page is still there for the next feature to hit. "Highest useful" includes "useful": a fix that hides a broken lower promise is not solving at the top, it is hiding the bottom.
- Descending on curiosity rather than signal. The request is fast, the handler is fast, and the engineer opens the query plan anyway because query plans are interesting. Learning is fine; it should not be labelled investigation.
- Checking a level's promise with the wrong observation. "The query is fast in my terminal" checks a different query on different data at a different time from the one in the slow request; the level is cleared falsely and the descent goes past the cause.
- Treating the rule as "always start at the user". Some problems are stated at a low level from the start — "fsync latency doubled after the firmware update" — and starting there is starting at the highest level *where the problem is stated*.
- Starting high and descending on signal is slower than a correct guess by an expert who has seen this exact problem; the rule trades the expert's best case for everyone's average case.
- Timing at each contract is instrumentation that may not exist yet, and adding it is work that the descend-by-intuition approach skips — until it needs it.
- A fix at the highest level is sometimes a product change ("search returns a page, not everything"), which involves people who were not in the room for a "performance bug".
- "Premature optimisation is the root of all evil, so never look at the database." The slogan, made precise: do not fix a level whose promise you have not observed to be broken. Once timing puts the time inside a query, looking at the database is not premature; it is the next step.
- "Highest level first means the frontend first." The highest level is the one where the problem is stated, and for "slow" that is the request boundary, which the frontend and the backend share. For "wrong total" it is the service. The user's screen is the top of the stack; it is not always the top of the problem.
- "If the fix works, the level was right." A cache in front of a query that reads every page works. The test is whether the fix is stated in the words of the cause, and a cache is stated in the words of the symptom.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe rule applies to any layered system — debugging, performance, correctness, and design decisions alike. A wrong-total bug, a slow search and a "should we shard?" question each have a highest level at which they can be stated and checked.
- TEAM-SPECIFICA solo engineer must do the whole descent. On a team with a database specialist, the rule decides when to hand the problem over — after the contract above the database has been checked — and the specialist's first question is usually whether that was done.
- CONTESTEDThe strongest opposing view: experienced engineers pattern-match straight to the likely level because the prior is strong — "search by name is slow" is an index nine times out of ten — and starting high every time throws away that prior and costs an hour per bug. That view is right when the prior is really that strong and the check at the guessed level is cheap and non-destructive; it goes wrong exactly in the cases where the guess is confident and wrong, which are the expensive ones, and the rule exists to bound that cost.
- ILLUSTRATIVEThe slow search, its two causes a month apart and the checkout under load are invented to show the descent-on-signal pattern; real timings and the number of levels vary.
Where the depth lives
This domain asks the question and hands the answer off by name.