Going One Layer Deeper
"Checkout is slow" → Backend → the database query → the B-tree → pages → the OS cache → the SSD. Each layer is a domain of its own, and the move is knowing when a layer's explanation is complete and when the next one down is required — never skipping a layer, and never going one further than the evidence sends you.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
When does a problem require going one layer deeper — and how do you know you have gone deep enough?
Checkout is slow. I found the slow query, added an index, and it is still slow. Someone says "check the buffer pool", someone else says "it might be the disk", and I have no way to tell whether those are the next step or a rabbit hole. Each layer down is a subject I have not studied.
Go all the way down. If the index did not fix it, the problem is deeper, so read about the buffer pool, then the page cache, then SSDs, until something explains it. Or, equally common: stop at the index, declare the database "just slow", and add a cache in front of it.
Going all the way down produces a week of reading and a theory about SSD write amplification for a query that reads. Each layer is a real subject and none was entered with a question, so each was learned as a tour and none was checked against the symptom.
- Going all the way down produces a week of reading and a theory about SSD write amplification for a query that reads. Each layer is a real subject and none was entered with a question, so each was learned as a tour and none was checked against the symptom.
- Stopping at the index produces a cache that hides a query which is slow for a reason that will return — the cache fills, the miss path is the same query, and the first sale day is the first cache-cold day.
- Because the layers are subjects the engineer has not studied, "one layer deeper" feels like a wall. The unknown becomes a blocker instead of a question, and the problem is handed off with "something in the database".
- The layers below the query are treated as one undifferentiated "the database is slow", so the evidence that would tell them apart — pages read from the pool versus from disk — is never collected, and the discussion is opinion.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Descend one layer at a time, and only when the current layer's promise has been checked and holds. The query is slow (request → backend → query: established). The index exists and the plan uses it (the index layer: holds). Symptom persists. So the layer below the index is next — pages — and not the SSD, which is two layers further and has not been earned (Highest Useful Level First).
- At each layer, ask the two-part question: what does this layer promise the one above, and what evidence would show that promise broken? The index promises "few pages touched per lookup"; the evidence is the page count. The buffer pool promises "hot pages are in memory"; the evidence is pool hits versus disk reads. The OS cache promises the same one floor down. The SSD promises "a block in a bounded time"; the evidence is device latency.
- Treat each layer as a domain with an entry question, not a subject to learn. "How does a buffer pool decide what to keep?" is a research question with a small answer (Researching an Unknown Technology); "learn database internals" is a semester. Enter the layer with the question the symptom gave you, get the answer, and either the layer explains the symptom or its promise holds and you go one further.
- Stop when the layer's evidence explains the symptom and a fix stated in that layer's words removes it. Record the layers you passed through and what held at each; the next slow-checkout investigation starts with that record, not from the top.
Every layer is a domain
The pipeline is the descent for "checkout is slow", layer by layer. Each step names the layer's promise and the observation that checks it; the failure column is what happens when the layer is skipped or entered without a question. Each layer is taught by a different part of the atlas, which is the point: this lesson cannot teach the buffer pool, and it does not need to — it needs to send you there with one question.
- 1Backend
promise: the handler and service finish within budget; observation: timing at each boundary puts the time in one query
fails by guessing the database before timing has placed the time there
- 2Database query
promise: the plan finds rows efficiently; observation: EXPLAIN on this query with real data
fails by adding an index without seeing the plan, then not checking the plan changed
- 3B-tree index
promise: few pages per lookup; observation: the buffer statistics for this query
fails by "the index exists" taken as "the index is used and is enough"
- 4Pages and buffer pool
promise: hot pages in memory; observation: pool hits versus disk reads for this table over the slow period
fails by skipped on the way to the disk, when it is the most common break
- 5OS page cache
promise: recently read blocks served from memory; observation: file reads hitting the cache versus the device
fails by confused with the database's own pool — two caches, two promises
- 6SSD
promise: a block read or sync in bounded time; observation: device latency, especially on sync
fails by entered first because it sounds fundamental; entered last because every layer above held
The descent stops at the first broken promise. Both worked examples used this pipeline and stopped at different rows.
What each layer promises, and who teaches it
The matrix is the reference to keep beside the pipeline: each layer, its promise, and the atlas domain that teaches how to observe and fix it. The right-hand column is the hand-off — the layer is a domain, and this lesson's job ends at the door.
| Layer | Promises upward | Broken looks like | Taught in |
|---|---|---|---|
| Service and query | one efficient query per checkout | many queries, or one that filters on the wrong column | Backend; this domain |
| B-tree index | a lookup touches few pages | a scan, or an index whose leading column is not the filter | Database: indexes; DSA: B-tree |
| Buffer pool | hot pages stay in memory | misses on a pool smaller than the working set | Database: buffer pool |
| OS page cache | recently read blocks are in memory | file reads going to the device on a machine short of memory | OS: file reads, page faults |
| SSD | reads and syncs in bounded time | commit waits on sync; latency spikes on a busy device | Hardware: storage path, memory hierarchy |
Deeper, or done?
The decision at every layer is the same and the criteria are the whole method. It is written as a device because the two reflexes — always deeper, never deeper — are each one of its options taken unconditionally.
At this layer, do I descend, fix, or stop?
when this layer's promise was checked with the right observation and holds, and the symptom persists
cost a new vocabulary and one more observation to collect; never more than one layer at a time
when this layer's promise is broken and the fix can be stated in this layer's words
cost the fix may be operational (size the pool) or design (change the query), and the second involves the layer above
when the fix removed the symptom and no remaining symptom sends you lower
cost the layers below stay unexplored, deliberately; write down where you stopped and why
when the break is at a layer you cannot change now — a managed database's pool, a shared device — and the symptom must go away today
cost a cache or a bigger machine that compensates for a named layer; the record says which, so the bandage is revisited
How to do it
Most important first.
- Write the layer stack for the slow path: request → handler → service → query → index → pages → OS cache → device. Mark the layers already cleared with the evidence that cleared them.
- For the next uncleared layer, write its promise and the one observation that would show it broken. Collect that observation and nothing else at this layer.
- If the promise holds, mark it and descend one. If it is broken, restate the problem in this layer's words and fix it here.
- When a layer is a domain you have not studied, write the entry question and go to the domain's lesson for that question — the zoom lab links each layer to its domain.
- Before adding a cache or a bigger machine at any layer, say which layer's promise it compensates for. If you cannot, it is a bandage over an unchecked layer.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Checkout is slow, index added, still slow. Cleared: request, handler, service (timing puts the time in one query); query (the plan uses the index). Next layer: pages. Promise: few pages per lookup. Observation: the query's buffer statistics show a small number of pages — the promise holds. Descend: buffer pool. Promise: those pages are in memory. Observation: nearly all reads are pool misses on a pool sized far below the working set. Broken here. Restated: "the hot pages of the orders table do not fit in the pool". Fix in this layer's words: size the pool to the working set. The OS cache and the SSD were never entered; they were two layers below the break.
- Same symptom, a different store. Pages: few. Pool: hits. Descend: OS cache — holds, the reads are served from memory. Descend: device — fsync on commit takes a long and variable time; the transaction's commit is waiting on the disk. Restated: "commit latency is bound by the device's sync time". The fix is at the device or the commit policy, and it was reached in four checked steps, each one earned, rather than by a guess at the bottom.
- The stop condition, applied. After the pool fix, checkout is within budget. Someone proposes "while we are here, tune the page cache". The layer below the fix has not been sent for by any symptom; the record says where the investigation stopped and why, and the tuning goes on the list of things to do when a symptom asks for it.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Each layer in your notes has a promise, an observation, and a verdict — holds or broken — and the descent stopped at the first "broken".
- The fix is stated in the words of the layer where the promise broke, and you can name the layers below it that were not entered and why.
- A layer you had not studied was entered with one question and left with one answer, and you can say what the entry lesson for it was.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Which layers of this path have been cleared, with what evidence — and which is the next one down?
- ?What does the next layer promise the one above, and what single observation would show that promise broken?
- ?What is the entry question for a layer I have not studied, and where does the atlas answer it?
- ?Has the symptom sent me to the layer I am about to enter, or am I going there because it is next?
What can go wrong
- Skipping layers. "The index did not help, so it must be the disk" jumps over pages, pool and OS cache, any of which could be the break; the evidence collected at the disk cannot say anything about them.
- Going one layer too far. The pool fix works, and the engineer continues to the SSD because it is interesting. Learning is welcome; labelling it as the investigation confuses the record.
- Entering a layer as a subject. "I need to understand the buffer pool" becomes a week of reading with no observation taken; the symptom did not need understanding of the pool, it needed one number from it.
- Clearing a layer with the wrong observation. "The query is fast in my terminal" at a different time on a warm pool does not clear the pool layer for the cold Friday-evening case.
- One layer at a time is slower than a correct guess at the bottom, and the guess is occasionally correct; the discipline trades the lucky case for a bounded one.
- Each layer entered is a domain's vocabulary to acquire, at least to the depth of one question; the descent is also a curriculum, and it takes the time a curriculum takes.
- The record of cleared layers is only valid for this symptom on this system today; it accelerates the next investigation only if it is kept, and it is rarely kept.
- "Deeper is more rigorous." Deeper is only more rigorous when the layer above has been checked. An SSD theory reached by skipping the pool is less rigorous than a pool measurement, however much lower-level it sounds.
- "If I don't understand the layer I can't investigate it." A layer is entered with one question and needs one observation. Not knowing the buffer pool is not a blocker; not having the entry question is (An Unknown Is Not a Blocker).
- "The database is just slow; add a cache." A cache compensates for a specific layer's broken promise — usually the pool or the query. Adding one without naming that layer is choosing not to know where the problem is, and the miss path will find out for you.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny layered path descends the same way: a slow model call through tokeniser, batch, GPU transfer, kernel; a failed build through package, resolver, registry, network. The promise-and-observation step per layer is the transferable part.
- STAGE-SPECIFICIn a prototype the descent usually stops at the query, because the layers below are a managed database's problem and the working set fits anywhere. In production with a real working set, the pool and device layers are where the Friday-evening symptoms live, and the record of cleared layers becomes an operational document.
- ILLUSTRATIVEThe two slow-checkout descents — one breaking at the buffer pool, one at device sync — are invented to show the stop condition at two different depths; real numbers and the layers that break vary per system.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's /manifesto/layers page walks the same descent — checkout to the SSD — as a philosophy of not delegating understanding; this lesson is the same descent as a working method, with a stop condition.