Debugging Is Problem Solving
A bug is an unknown with a symptom attached. Symptom → observation → hypothesis → experiment → evidence → updated hypothesis is the same loop the domain uses for everything else, and it turns "Payment failed" into a sequence of questions each of which can be answered.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
The checkout page says "Payment failed" and you have no idea why. How do you make progress without knowing where the bug is?
A customer reports that checkout shows "Payment failed". You click through it yourself and see the same message. The frontend, the backend, the payment provider and the database are all plausible, and staring at the message tells you nothing about which one it is.
Open the code where the message is rendered and read outward from there, changing anything that looks suspicious. It feels productive because you are in the code, and the code is where bugs live.
The render site says if (!result.ok) show("Payment failed"). That is true of every possible cause — provider declined, backend timed out, request never sent, order row missing — so reading it narrows nothing, and the next thing to read is chosen by proximity, not by evidence.
- The render site says
if (!result.ok) show("Payment failed"). That is true of every possible cause — provider declined, backend timed out, request never sent, order row missing — so reading it narrows nothing, and the next thing to read is chosen by proximity, not by evidence. - Suspicious-looking code gets edited. The edit changes behaviour in some other way, the symptom persists or moves, and now there are two things to explain instead of one.
- The cause is assumed from a previous bug. "Last time it was the provider" becomes the hypothesis without any observation to support it, and an afternoon goes into the provider dashboard while the backend logs, which would have settled it in a minute, stay unread.
- After a few hours the system has been restarted, redeployed and had three files edited, and the honest status is "it might be fixed". Nothing was learned, so it will come back.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Treat the bug as an unknown, not as a location. "Payment failed" is a symptom; the question underneath it is "which component decided the payment failed, and on what basis?" — and that question can be answered with observations before any code is changed.
- Run the loop deliberately. Observe: what exactly happened, where, with what input? Hypothesise: name a cause that would produce exactly that observation. Predict: if the hypothesis holds, what else must be true that you have not yet looked at? Experiment: go and look. Update: the hypothesis survives, dies, or splits into two.
- Choose experiments by how much they distinguish. An observation every hypothesis predicts equally — "the error appears every time" — is worth nothing; the one that separates "backend never called the provider" from "provider declined" is worth everything. Cheap, distinguishing observations first; code changes last.
- Keep the hypothesis written down. It is the thing you are testing, and the moment it becomes implicit you are back to changing things and hoping.
The loop, pointed at a bug
The problem-solving loop this domain uses everywhere — understand, sharpen the unknown, experiment, observe, update — is what debugging is. The only difference is that the system already exists, so observations are cheap and plentiful, and the discipline is choosing them rather than finding them.
Each step has a characteristic way of being skipped, and the pipeline names it. Most bad debugging sessions are not bad reasoning; they are a step left out, usually the hypothesis or the prediction.
- 1Symptom
State precisely what was observed, by whom, with what input, against what expectation.
fails by Working from the reported wording ("it's broken") instead of the observation.
- 2Observation
Gather what the system already knows: network tab, request log, provider dashboard, database row.
fails by Going to the code before looking at what the running system says.
- 3Hypothesis
Name a cause that would produce exactly this observation — written down, and more than one if more than one fits.
fails by Carrying the last bug's cause forward as this bug's explanation.
- 4Prediction
Say what else must be true if the hypothesis holds, before looking.
fails by Looking first and then explaining whatever was found as confirmation.
- 5Experiment
Make the cheapest observation that distinguishes the surviving hypotheses. A code change is one option, rarely the first.
fails by Changing code because it is the experiment the editor makes easiest.
- 6Update
Cross off what the evidence contradicts; sharpen what survives; loop.
fails by Keeping a favourite hypothesis alive after the evidence killed it.
The loop terminates when one hypothesis survives and the fix that follows from it removes the symptom for a reason you can state.
"Payment failed" as an unknowns board
Written as the domain writes any unknown, the bug stops being a feeling and becomes a list of questions with experiments. Notice that each experiment is an observation of the running system, and that none of them requires changing a line.
- ✓The message appears for every card since this morning, on our checkout page, in test mode and in production.
- ✓The path is browser → our API → payment provider → our database, in that order.
- ✓A cart refactor was deployed last night; nothing else changed on our side.
- ~The provider is up — to be checked on their status page rather than believed.
- ~The message means the provider declined — to be checked; our frontend shows the same text for any non-OK response.
? Something is wrong with payments.
becomes Which component first decides that the payment failed — browser, our API, the provider, or the database write afterwards?
experiment Open the network tab and read the status code and timing of the request to the pay endpoint; then check whether the provider dashboard shows an attempt at that time.
? Maybe the provider changed something.
becomes Did our backend send a request to the provider for this checkout at all?
experiment Search the backend log for the request id from the browser; look for the outbound call to the provider and its response, or its absence.
? It could be the deploy.
becomes Does the same checkout succeed on the build from before last night's cart refactor?
experiment Run the previous build locally against the same test card; if it succeeds, the cause is inside the diff (Binary Search Over the System).
Three unknowns, three observations, none of them a code change. Whichever answers first collapses the others.
Where the observations come from
The layers between the click and the message are also the places where evidence lives. Each node in the diagram has something it can tell you and something it cannot: the browser knows the status code and the timing but not why; the backend log knows what the handler did but not what the provider saw; the provider dashboard knows about attempts that reached it and nothing about those that did not.
Debugging across those layers is the subject of the lab: gather evidence at each node, see which causes stay consistent with all of it, and notice what the button that changes something at random does to your knowledge — nothing.
How to do it
Most important first.
- Write the symptom precisely before doing anything: who saw what, when, with what input, and what they expected instead. "Payment failed" becomes "a card that succeeds in the provider's test dashboard shows Payment failed on our checkout page, every time, since this morning".
- List the components between the input and the symptom, and the observation each one could give you: browser network tab, backend request log, provider dashboard, database row. Each is an experiment that costs seconds (Inside and Outside the System).
- Write two or three hypotheses that would each produce the symptom, and for each, one observation that would be true only if it holds (Experiment Design).
- Predict the observation before you make it — see Predict Before You Look — then make it, and cross off every hypothesis it contradicts.
- Change code only when a hypothesis has survived contact with evidence and the change is the test of it. Then verify the symptom is gone *and* that the evidence trail says why (Prediction Before Execution).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Symptom: "Payment failed" on checkout for every card since this morning. Components: browser → our API → payment provider → our database. First observation, the browser network tab: the request to
/checkout/payreturns a 500 in a fraction of a second. That already kills "provider declined the card" — a decline is a 4xx with a reason, and it would take longer than that. - Surviving hypotheses: the backend throws before calling the provider; the backend calls the provider and cannot parse the response; the database write after payment fails. Distinguishing observation: the provider dashboard shows no attempts since this morning. Now only the first survives: the backend never reaches the provider.
- Backend log for the request:
TypeError: cannot read property "id" of undefinedin the handler, at the line that reads the cart. Hypothesis sharpened: something changed about the cart shape this morning. The deploy log shows a cart refactor merged last night. The fix is one line; the point is that it was found by three observations, none of which involved editing code. - Written down afterwards: the symptom, the three hypotheses, the two observations that decided it, the cause and the fix — so that the next "Payment failed" starts from a list rather than from nothing (The Engineering Notebook).
How you know it worked
What now exists that did not before, and what question you can now ask.
- There is a written hypothesis, and you can say which observation would kill it.
- Every observation you make removes at least one candidate cause; none of them is made just to feel busy.
- The layers between input and symptom are listed, and you know which one you have not yet looked at.
- When the fix lands, you can explain the failure to a colleague as a chain of evidence, not as "I changed X and it went away".
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What exactly is the symptom — who saw what, with what input, and what did they expect?
- ?Which components sit between the input and the symptom, and what observation can each give me?
- ?What are two causes that would produce exactly this, and what observation separates them?
- ?Which observation is cheapest and most distinguishing, and have I made it yet?
- ?If my hypothesis is right, what else must be true that I have not looked at?
What can go wrong
- The loop becomes a ceremony. Five hypotheses are written for a bug whose cause is visible in the first log line. When one observation settles it, stop and fix it.
- Hypotheses are formed but never tested; the engineer argues about which is most likely instead of looking. An observation that costs a minute beats a debate that costs an hour.
- The loop runs on the wrong symptom. The reported message is "Payment failed" but the actual failure is that the order was created twice; debugging the message fixes the wording and leaves the double charge (Duplicate Requests).
- The evidence is gathered and then ignored because it contradicts the favourite hypothesis. The loop only works if the hypothesis is allowed to die.
- Writing the symptom and hypotheses down costs minutes at the moment when the pressure to act is highest, and on a bug that turns out to be trivial those minutes were wasted.
- Evidence-first debugging depends on evidence existing. In a system with no logs and no provider dashboard, the first experiment is adding observability, which is slower than guessing and occasionally slower than a lucky guess.
- The discipline is hard to keep at 2 a.m. during an incident, which is exactly when it matters most — see how Debugging an Incident in Progress structures it for a team.
- "So never touch the code until you have a hypothesis." Reading code is an observation and often a cheap one. The rule is against *changing* code as a way of finding out, not against reading it to form a hypothesis.
- "Debugging is about tools." Tools produce observations; the loop decides which observation to make. A debugger attached without a hypothesis shows you a thousand true facts and no way to rank them.
- "This is what senior engineers do." It is what they do in their heads, fast, which is why it looks like intuition. The intuition is a large stock of hypotheses and a habit of checking the cheapest one first.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe loop is the same for a frontend layout bug, a flaky test, a slow query and a production incident; only the set of available observations changes.
- TEAM-SPECIFICSolo, the hypothesis can live in your head for a short bug. On a team, or during an incident, it must be written where others can read it, because the person making the next observation may not be you.
- ILLUSTRATIVEThe store, the cart refactor, the 500 and the empty provider dashboard are invented to show the shape of the loop; no real incident is described.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's "debug without AI" practice at /manifesto/without-ai is this loop with the tool removed: the hypothesis has to be yours before the assistant is asked to check it.