The Question Quality Ladder
A question is vague, researchable or decisive, and the difference is checkable: does it name a component, name a boundary, ask for a mechanism, scope itself to my system, and ask for a decision? Five criteria, applied to your own questions before you send them anywhere.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
How can you tell, before you spend an hour on it, whether a question is one that can be answered — and whether its answer will decide anything?
You keep a list of things you do not understand about the store. It has eleven entries and most of them are single words or "how does X work". You have tried to research the top three and each one turned into a browsing session with no end. You suspect the questions are the problem, not the answers, but you cannot say what is wrong with them.
Research harder. Pick the entry that worries you most, open more sources, take notes, and trust that enough reading will eventually turn "caching" or "auth" into something concrete. Effort feels like the missing ingredient.
The notes grow and the list does not shrink. "Caching" is now a page of notes about caching, and still nobody could say what the store would do differently because of it.
- The notes grow and the list does not shrink. "Caching" is now a page of notes about caching, and still nobody could say what the store would do differently because of it.
- The AI is asked the eleven questions in turn and answers all eleven fluently. Fluent answers to vague questions are the most convincing form of no progress: the transcript looks like a tutorial and contains no decision.
- The list cannot be prioritised, because every entry is the same size — one word. Which of "caching", "auth" and "inventory" matters first is unanswerable until each is a question with a consequence.
- When one entry is finally answered, it is answered for the tutorial's system rather than yours, because nothing in the question said which system it was about.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Grade the question before you research it, against five checkable criteria. Does it name a component (a part of some system, ideally yours)? Does it name a boundary (something crossing between two parts, or between your system and the outside)? Does it ask for a mechanism (a sequence, a rule, a data flow) rather than a concept? Is it scoped to your system, your case, your data? Does it ask for a decision — would the answer choose between designs?
- Read off the grade. None of the five: vague — a feeling with a question mark. Component, boundary and mechanism, scoped to you: researchable — a page of documentation or a test call can answer it. All five: decisive — the answer changes what you build. The criteria are not a score to maximise; they are the list of what to add next.
- Add the missing criterion, in roughly that order, and re-grade. The order matters because a boundary is hard to name without a component, and a decision is hard to ask for without a mechanism. Most vague questions become researchable by adding a component and a mechanism, and researchable questions become decisive by asking who owns the fact.
- Research only researchable questions, and decide only on decisive ones. A vague question sent to documentation returns the table of contents; a researchable question returns a page; a decisive question returns a page and a design.
Five criteria, three grades
The matrix is the grading rubric. Each row is a criterion; each column is what the criterion looks like when present, and what its absence does to the question. The grades fall out of counting, but the point of the rubric is the right-hand column: what to add.
The order of the rows is the order in which criteria are usually added. A component makes a boundary nameable, a boundary makes a mechanism askable, scope makes the mechanism yours, and only a mechanism that is yours can be decided.
| Criterion | Present when the question… | Absent, the answer is… | Add it by asking… |
|---|---|---|---|
| Component | names a part of a system — my backend, the provider, the product page | about everything, so about nothing | who or what is involved? |
| Boundary | names something crossing between parts — a request, a callback, a row | an essay about a concept | what passes between them, in which direction? |
| Mechanism | asks for a sequence, a rule or a flow rather than "how it works" | a definition | what happens, step by step, when…? |
| Scope | says "in my system", "when my customer…", "with my data" | true of the tutorial's system and untested on yours | in my case specifically…? |
| Decision | would choose between designs depending on the answer | interesting and inert | what would I build differently if the answer were X rather than Y? |
One question, graded at each rung
The search question from the worked example, laid out as a ladder. The vague rung meets no criterion. The better rung names components and a mechanism and is scoped, so it is researchable — a pattern query against the real catalog answers it in minutes. The best rung adds the decision and the constraint that makes the decision possible.
The why line is the test the whole lesson is built on: what does the best form make answerable that the others do not?
why The best form is answerable by a decision with a stated reversal condition: the database for V1, the engine when the catalog grows, needs relevance ranking, or spans languages. The vague form is answered by a list of search technologies; the better form by a measurement that, on its own, does not say what to do.
How the grading itself goes wrong
The ladder is a tool for the person asking, and it fails in the ways tools fail: applied mechanically, applied to the wrong object, or applied to other people. The table names the failures the assessor cannot catch, because they are failures of judgment rather than of wording.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A long question is graded decisive because it has many clauses | Research produces a page and no design changes | Length was mistaken for a decision; nothing in the question would choose between two builds | Write the two candidate designs next to the question; if you cannot, it is researchable at best |
| The component named does not exist in your system | The decisive answer applies to a design you do not have | The climb invented the architecture instead of reading it | Name only components that are in the code or the requirements; a proposed component is an assumption, and goes on the assumed list |
| A fact lookup is forced into a decision | Time spent finding a "choice" in a question with one answer | Treating decisive as the only respectable grade | Look the fact up; researchable is a grade, not a deficiency |
| The ladder is used to grade a colleague's question in review | They stop asking | The rubric turned into a verdict | Grade together and add the missing criterion with them — the move is collaborative or it is not the move |
How to do it
Most important first.
- Take the vaguest entry on your list and mark, in the margin, which of the five criteria it meets. Most first drafts meet none; that is information, not failure (Question Quality).
- Add a component first — replace "caching" with "the product page and the database behind it". Then a mechanism — "what happens on each request". Then scope — "when a customer opens a product page on my store".
- Ask whether an answer would change the design. If it would not, ask what fact about the system you are actually worried about and put that fact in the question.
- Paste the result into the unknowns board's assessor and read its reasons, not its verdict. The reasons name the criterion you are still missing (The Unknowns Board).
- Attach an experiment to every researchable question and two candidate designs to every decisive one. A researchable question with no experiment is a reading assignment; a decisive question with no candidates is not yet decisive.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- "Caching?" — meets no criterion. Add a component and a mechanism: "what does the product page fetch from the database on each request, and how often does that data change?" Researchable: one query log and one look at the admin's edit frequency answer it. Add a decision: "should the product page read prices from the database on every request, or from a copy that admin edits refresh?" Decisive — and the answer, for a store where prices change a few times a day, is the database, which means the "caching" entry disappears from the list rather than being researched.
- "Auth?" — no criterion. Climb: "which requests in my store must know who the customer is, and which must not?" — component (requests), boundary (identified vs anonymous), scope. Researchable by listing the endpoints. Decisive form: "should the cart belong to a session or to an account, given that guest checkout is required?" The answer decides the cart's data model, which is why it belonged near the top of the list, and why "auth" as one word could never have told you that.
- "Search?" — climb to "when a customer types part of a product name, should matching happen in the database with a pattern query or in a separate search engine, given that the catalog has one language and a few thousand products?" The scope clause is what makes it decisive: it rules out the engine for V1 and says exactly which requirement change would rule it back in (Researching an Unknown Technology).
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every entry on your list has a grade, and you can say which criterion each is missing.
- The list has an order, because decisive questions with design consequences are visibly heavier than researchable ones with a page to read.
- Some entries vanished without being researched, because the decisive form made the answer obvious from what you already knew.
- The questions you send to documentation, search or an AI come back with short answers you can check.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Which of the five — component, boundary, mechanism, scope, decision — is this question missing, and which is easiest to add next?
- ?Is this a question I need to research, or a question I need to decide — and do I have an experiment or two candidates ready for it?
- ?Which entries on my list would disappear if I wrote their decisive form, because I already know the answer?
- ?What fact about my system am I actually worried about, and is that fact in the question?
What can go wrong
- Grading as a ritual. Five ticks are counted and the question is declared decisive because it is long. Length is not a criterion; the test is whether an answer would change the design, and a long question can still fail it.
- Demanding decisive when researchable is enough. "Which signature scheme does the provider use?" has no design to choose between; it is a fact to look up. Forcing a decision onto it wastes the climb.
- Climbing without checking. The component and mechanism are invented — "the cache in front of the database" for a store that has no cache — and the decisive question is about a system that does not exist. The criteria say "name a component", and the component has to be real.
- Applying the ladder to someone else's question in a review and stopping there. Grading a colleague's vague question is easy and unkind; helping them add the missing criterion is the move.
- Grading every question is overhead, and on a familiar problem it is overhead you do not need: an engineer who has built the thing before grades in their head and the written ladder would slow them down.
- The five criteria favour questions about systems with parts. A question about an algorithm ("is this greedy choice safe?") maps awkwardly — the "component" is a constraint, the "boundary" an input range — and the ladder is a looser fit.
- Scoping a question to your system hides answers that would have been useful later. The researchable form of "search" rules out the search engine correctly for V1 and teaches you nothing about it; that is the intended saving, and it is a saving of understanding as well as time.
- "A question needs all five to be worth asking." No. Researchable — the middle grade — is where most useful questions live, and most of an engineer's day is answering them. Decisive is for the handful that choose a design.
- "The assessor decides whether my question is good." The assessor applies the same criteria mechanically and says which it could not find. It is a mirror, not a judge; a question the assessor grades decisive can still be about the wrong thing.
- "Vague questions are a sign of not knowing enough." They are a sign of not having asked the ladder yet. The person who knows the most about payments still starts with "hm, payments" — they just climb faster.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe five criteria grade any technical question; what changes across domains is what counts as a component (a service, a function, a constraint, a tensor) and what counts as a decision.
- SIMPLIFIEDThree grades and five criteria are a teaching model. Real questions sit between grades, a question can be decisive for one reader and merely researchable for another who already knows the answer, and the assessor's heuristics are word-level approximations of the criteria, not the criteria themselves.
- ILLUSTRATIVEThe eleven-entry list, the few-thousand-product catalog and the few-times-a-day price changes are invented to give the worked examples a shape; a different catalog would grade the search question differently.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's "Review the LLM's answer" route at /manifesto/review is the ladder applied after the fact: a fluent answer to a vague question is the thing it teaches you to notice.