Experiment Design
Question, hypothesis, test, observation, conclusion — the same structure whether the experiment is a load test, a spike or a debugging probe. The hypothesis is written before the test so the result can disagree with you; the observation is recorded before the conclusion so the conclusion cannot rewrite it.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You are about to try something to find out whether it is true. How do you set it up so that the result can actually change your mind?
Product search on the store feels slow. You think it is the query — a name search with a leading wildcard that cannot use the index. You are about to add a full-text index and see if it helps. A colleague asks "how will you know?" and you realise the honest answer is "it will feel faster", and that you would have declared victory either way.
Try the fix and see. Add the index, reload the page, notice it feels better, merge. It is fast and it is action, and the feeling of improvement is genuinely persuasive — that is the problem with it.
The result cannot disagree with you. There was no number before, so there is no number after, and "feels faster" is true of any change made by someone who expects it to help.
- The result cannot disagree with you. There was no number before, so there is no number after, and "feels faster" is true of any change made by someone who expects it to help.
- The wrong thing gets credit. The page was slow because of an image that loaded before the results, the index changed nothing, and the index is now cargo in the schema with a commit message that says it fixed search.
- The experiment is not repeatable. Someone asks in a month why the index exists and whether it still helps; there is no way to re-run what was done because what was done was "reload and see".
- Each experiment is one-off, so nothing accumulates. Five fixes later, search is faster and nobody can say which of the five did it or whether three of them could be removed.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Write five things, in order, and do not skip one. Question: what do I want to know, precisely — "is the leading-wildcard name query the reason search takes as long as it does?" Hypothesis: what I expect, as a claim that could be wrong — "the query is most of the page time; with a full-text index it will drop by most of that." Test: what I will do and what I will measure — "time the query and the full page, ten searches each, before and after, same data, same machine." Observation: what I saw, as numbers, written before I think about what they mean. Conclusion: what the observation says about the hypothesis, and what it does not.
- The hypothesis is written *before* the test so that the test can refute it. A hypothesis written afterwards always matches. The same goes for the measurement: decide what you will measure and what would count as "yes" before you look.
- Control what you can. Same data, same machine, same query set, one change at a time. A test that changes two things and improves has learned nothing about either.
- Record the observation as it was, separately from the conclusion. Numbers first, meaning second. When the numbers disagree with the hypothesis, that is the experiment working, and the conclusion is "the query was not the cause; the next candidate is…" (Debugging Is Problem Solving).
The five steps, and how each one lies
The pipeline is the whole method, and its value is in the order. The failure column is the reason the order matters: each step has a way of being done after the next one, and each of those ways produces an experiment that confirms whatever you already believed.
- 1Question
What I want to know, precisely enough that an answer would be recognisable
fails by "Why is search slow?" — a question with no measurable form
- 2Hypothesis
What I expect, with a size or direction, written before the test
fails by Written after the result, or without a number any result could contradict
- 3Test
What I will change, what I will measure, what counts as yes — plus the baseline
fails by No baseline; two changes at once; a warm cache against a cold one
- 4Observation
The raw result, recorded before interpretation
fails by Summarised into what the hypothesis wanted; outliers dropped
- 5Conclusion
What the observation says about the hypothesis, and what it does not show
fails by Overreach: "search is fixed" from one machine, one data size
When the observation refutes the hypothesis, the conclusion names the next hypothesis. The harness is reused; the next experiment costs minutes.
The record, as it would sit in the notebook
This is the experiment written down — short enough to keep, complete enough to re-run. The observation section is deliberately raw; the conclusion is deliberately smaller than the hypothesis was hoping for.
1QUESTION Is the leading-wildcard name query the reason product search2 takes as long as it does?3HYPOTHESIS The query is most of the request time. A full-text index4 removes most of that. (Written before measuring.)5TEST Log query time and total request time for the same ten6 searches, dev DB with a production-sized products table,7 before and after the index. Yes = query time drops by most8 of its value AND request time drops by about as much.9BASELINE query: small fraction of request. request: dominated by10 ten sequential image-metadata calls, one per result.11OBSERVATION Hypothesis refuted at baseline. Index not added.12 Raw timings kept in search-exp-1.csv (ten rows, per stage).13CONCLUSION The query is not the bottleneck at this table size.14 Does not show: behaviour at a larger table; production15 network for the image call.16NEXT H2: batching the image fetch into one call removes most of17 the request time. Same harness, one change.The line that matters most is "Written before measuring". The second is "Does not show". Everything else is bookkeeping that makes those two honest.
Which experiment, for which uncertainty
The same five steps wear different instruments. The decision names the common shapes; the criteria are what the question is about, and the cost is what each instrument leaves unproven. Choosing the wrong instrument is the most common way a well-structured experiment answers the wrong question.
What is the uncertainty about?
when The question is "how long" or "how many" — search latency, checkout under concurrent orders.
cost Only as true as the data size, machine and cache state; the conclusion must name them.
when The question is "what does it do" — does the callback race the response, does the browser resume the upload.
cost Proves the behaviour for the cases tried; the untried cases are the doesNotProve list.
when The question is "why" — which of several candidate causes is consistent with the evidence.
cost Each probe rules candidates out rather than confirming one; the conclusion is a shortlist until one remains.
when The question is "is it better" with no single number — search relevance, an AI assistant's answers.
cost The set is the measurement; a small or biased set produces confident nonsense, and "one change at a time" is hard to honour.
How to do it
Most important first.
- Write the question and the hypothesis in the notebook before touching anything, and include the prediction as a number or a direction: "most of the page time", "at least half", "no change". A vague hypothesis cannot be refuted (Prediction Before Execution).
- Choose the measurement to match the question. "Is the query the cause?" needs the query time *and* the page time; the query alone cannot say how much of the page it is.
- Get the baseline first. A before-number is the most commonly skipped step and the one without which nothing else means anything (Logs Are Evidence, Not Thinking).
- Change one thing. If two changes are tempting, run two experiments; the second takes minutes once the first is set up.
- Record observations raw — the ten timings, not their average — and conclude afterwards. Write what the result does *not* show: "on this data size; on this machine; for name search only."
- Keep the record. An experiment that can be re-run in a month — same script, same query set — is the difference between knowledge and a memory (The Decision Journal).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Question: is the leading-wildcard name query why product search takes as long as it does? Hypothesis: the query is most of the page time, and a full-text index would remove most of that. Test: log the query time and the total request time for the same ten searches, on the development database with the production-sized product table, before and after adding the index. Success looks like: query time drops by most of its value and page time drops by about the same amount.
- Baseline, observed: query time is a small fraction of the request; most of the request is a call that fetches each result's image metadata one product at a time. Ten results, ten calls. The hypothesis is refuted before the index is added — the query was never the cause.
- Conclusion: the query is not the bottleneck at this data size; the per-result image call is. Does not show: whether the query becomes the bottleneck at a larger table (a second experiment, with the table grown), or whether the image call is slow in production the same way (the development machine's network differs). Next hypothesis: batching the image fetch into one call removes most of the request time. Same test harness, one change, run.
- The index was never added, and the record says why. In a month, when the product table has grown and search feels slow again, the same harness is run first, and the numbers say whether it is now the query — instead of someone remembering that "we tried an index once".
How you know it worked
What now exists that did not before, and what question you can now ask.
- The hypothesis is in the notebook with a timestamp earlier than the result, and it includes a predicted size or direction.
- There is a baseline number, and the after-number is compared to it rather than to a feeling.
- At least one experiment has refuted its hypothesis, and that was recorded as a finding rather than as a failure.
- An experiment from last month can be re-run today with the same script and compared.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What exactly do I want to know, and what do I expect to see — as a number or a direction — before I look?
- ?What will I measure, and what would the measurement look like if I am wrong?
- ?What is the baseline, and did I take it before changing anything?
- ?What does this result not show, and what is the next hypothesis?
What can go wrong
- The structure becomes ceremony. Five headings for every trivial change — "hypothesis: the typo fix will fix the typo" — and the discipline is abandoned because it is silly. It is for changes whose effect is uncertain.
- The hypothesis is written to be unfalsifiable: "the index will help." Help how much? Compared to what? A hypothesis that any positive number confirms is not one.
- The observation is edited by the conclusion. Three fast timings and seven slow ones become "mostly faster"; the raw ten would have said "bimodal — something else is going on".
- The controls are skipped because the result was so clear. A dramatic improvement on a warm cache, against a cold baseline, is the classic; the second run tells the truth.
- Writing the hypothesis and taking a baseline costs time on every experiment, and on the ones where the fix was obvious it looks like bureaucracy.
- A refuted hypothesis means the fix you were about to make is not made, which can feel like a lost afternoon to a stakeholder who wanted search faster today.
- Controls narrow what the experiment says — "on this machine, this data size" — and the honest conclusion is smaller than the confident one would have been.
- "This is for performance work." It is for anything whose effect is uncertain: whether a retry policy reduces failed orders, whether a cache changes the error rate, whether a prompt change improves the AI assistant's answers. The structure is the same; only the measurement differs.
- "Measure first" means "never act without a benchmark." It means: before a change whose *effect you are claiming*, have a number the claim can be checked against. A typo fix needs no baseline; a "this makes search fast" commit does — that is the slogan made precise.
- "A refuted hypothesis wasted the experiment." It is the most valuable outcome: an index not added, a cause found, a next hypothesis that is better than the first. Experiments that only ever confirm are not experiments.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALQuestion, hypothesis, test, observation, conclusion is the shape of any attempt to find out whether something is true, from a query index to a retry policy to a model prompt; what changes is the instrument and the number, not the order of the five.
- ILLUSTRATIVEThe ten searches, the ten per-result image calls, the "small fraction" and the development machine are invented to show an experiment refuting its own hypothesis; no real search latency is being quoted.
- DOMAIN-SPECIFICIn performance work the observation is a timing and controls are machine, data and cache state; in ML or an AI assistant the observation is a score over a held-out set and the control is the set itself — the structure holds, but "one change at a time" is far harder when the change is a prompt and the measurement is a judgement.
Where the depth lives
This domain asks the question and hands the answer off by name.