Counterexample Thinking
When a claim sounds right — "this always works", "that can never happen" — try to construct one case where it does not. A counterexample found in five minutes on paper is worth more than a week of confidence, and failing to find one after honestly trying is evidence, not proof.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Someone — possibly you — says "this always works" or "that can never happen". How do you test the claim before the system does it for you?
A colleague reviews your checkout and says "the order total can never be wrong, we compute it from the line items". It sounds right. It is also exactly the kind of sentence you have heard before a bug. You are not sure how to argue with it without sounding difficult, and you do not have a concrete objection — just a feeling.
Accept it, or argue about it in general. "Well, what if something changes?" — "Like what?" — "I don't know, something." The conversation ends where it started, and the claim stands by default because nobody produced a case.
The claim is tested by intuition, and intuition has a bias towards the happy path because that is what was built. "We compute it from the line items" is true of the code that was written and silent about the code that was not.
- The claim is tested by intuition, and intuition has a bias towards the happy path because that is what was built. "We compute it from the line items" is true of the code that was written and silent about the code that was not.
- Disagreement without a case is noise. The reviewer is not wrong to ask for one; a feeling cannot be checked, and the meeting moves on.
- The claim becomes an invariant nobody wrote down. Six months later a coupon feature multiplies a line total and the "never wrong" total is off by a rounding error on every third order, and the person who said "never" has left.
- "Always works" is treated as a property of the code rather than of the assumptions around it, so the assumptions are never listed and never checked.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Treat every universal — "always", "never", "can't", "guaranteed" — as a claim that a single case could refute, and then try, in good faith, to build that case. Not "what if something changes" but "here is a specific input, a specific sequence, a specific moment; walk it through and show me the total is right".
- Generate candidates from the places universals usually hide: the assumptions the claim depends on (the line items are correct, the prices did not change, the currency is one), the inputs varied to their edges (Edge Cases From Examples), time (what happened between two steps), and failure (a step that half-completed). Each of those is a factory for counterexamples.
- When you find one, you have not won an argument; you have found a requirement or an invariant that needs a mechanism. "The total is wrong when the price changes after add-to-cart" is a design question about snapshots, and the claim is rewritten as "the total is always the sum of the line prices *captured at order time*" — narrower, and now true.
- When you cannot find one after an honest search, say so precisely: "I tried price changes, rounding, deleted products, and concurrent edits and could not break it." That sentence is evidence. It is not proof, and it tells the next person where you looked.
Doubt versus a case
The difference between a review that helps and one that irritates is almost entirely whether the doubt comes with a case attached. The pair below is the same concern expressed both ways; only one of them can be answered, and only one of them changes the design.
"Are we sure the total can never be wrong? Feels risky to compute it on the fly."
"Alice adds a lamp at 40. Admin changes it to 45. Alice pays. What total is on the order, and what total was on the cart page she saw? Walk me through it."
The doubt can only be answered with reassurance, and reassurance is what produced the claim. The case has to be traced, and the trace either shows the total is fine — in which case the reviewer has learned where the snapshot happens — or shows there is no snapshot, which is a design decision and not an argument.
Where counterexamples come from
Counterexamples are constructed, not waited for, and they come from a short list of factories. The claim and its assumptions are written down first; then each factory is applied. The pseudocode is the search made explicit — a checklist in the shape of a program, run by hand.
1claim: for every order, at every time: total == sum(price * qty over lines)2assumes: price stable between add and pay; qty integer >= 0; one currency; exact rounding3 4for factory in [assumptions, edges, time, failure, concurrency]:5 case = construct(factory, claim)6 trace(case)7 if total differs from what should happen:8 record counterexample; narrow claim; name the mechanism9 else:10 record "tried: " + case11 12# assumptions → price changed after add: total 45, page said 40 → COUNTEREXAMPLE13# edges → qty 0, qty 1, qty 40 lines → holds14# time → product deleted between add and pay → COUNTEREXAMPLE (which price?)15# failure → payment succeeds, order write fails → total irrelevant; different invariant16# concurrency → line added mid-computation → holds (single read)Two of five factories produced a case; both turned into the same requirement — capture the price at order time — and the claim was narrowed to something true. The three that held are recorded, so the next reviewer does not repeat them.
What to do with a counterexample
A found case has three possible fates, and choosing between them is a decision about the product rather than about the code. The device below names the options; the lesson is that "fix it" is only one of them and often not the right one.
The case breaks the claim. Does the product care?
when The case is real and users would notice — a price change between add and pay changes what they are charged.
cost A design change (a snapshot column, a captured price) and a test that encodes the case.
when The case is real and nobody is harmed — a one-cent rounding difference on a three-line order.
cost A known edge in the notes and an assumption that has to be revisited if money handling becomes stricter.
when A rule elsewhere prevents it — the cart is locked during payment, so mid-computation edits cannot occur.
cost The claim now depends on that rule, which must be listed as an invariant, because removing the lock later silently reopens the case.
How to do it
Most important first.
- Rewrite the claim as a formal-ish sentence with its quantifier visible: "for every order, at every time, total = sum(line price × quantity)". The words "every" and "at every time" are where to attack.
- List the assumptions the claim needs to be true. Each assumption is a candidate counterexample when it fails: prices are stable, quantities are integers, one currency, line items are never edited after creation.
- Try the cheap counterexamples first: the edge axes, then time (something changed between two steps), then failure (something half-happened), then concurrency (two things happened at once).
- When a counterexample works, narrow the claim until it is true and note what mechanism makes it true (Finding Invariants From Examples). "Never wrong" becomes "never wrong given a snapshot at order time", and the snapshot is now a requirement.
- When it does not work, record where you looked. "Could not break it" with a list is a review comment; "looks fine" is not.
- Apply the same move to your own claims before a reviewer does, and to any slogan you catch yourself repeating — "just ship it", "measure first", "don't over-engineer" — until it is precise enough to have a counterexample.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Claim: "the order total can never be wrong; we compute it from the line items." Quantified: for every order at every time, total equals the sum over lines of price × quantity. Assumptions: price is the price at the time the total is read; quantities are non-negative integers; one currency; rounding is exact.
- Counterexample from time: Alice adds a lamp at 40, the admin changes it to 45, Alice pays. The total is computed from the current price, so it is 45; the confirmation email, generated from the cart page she saw, says 40. Which is "right"? Neither is *wrong* — the claim was ambiguous, and the counterexample turns it into a decision: capture the price at order time. The claim is rewritten and a snapshot column exists.
- Counterexample from arithmetic: three items at a price that produces a third of a cent each. Sum of rounded lines differs from rounded sum by one cent. Not a bug the founder cares about today; written down as a known edge, because "never" now has a footnote.
- Attempted counterexample from concurrency: two tabs edit the cart while the total is computed. Tried line added mid-computation, line removed mid-computation. Both produce a total that matches *some* consistent cart, because the total is computed inside one read. Could not break it; that is recorded, along with what was tried.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every "always" and "never" in the design or the review has either a counterexample next to it or a note saying what was tried.
- Claims have narrowed. "The total is never wrong" became "the total is always the sum of the prices captured at order time", and the capturing is a mechanism you can point at.
- Reviews contain cases instead of feelings. "What about a price change between add and pay?" replaces "are we sure about this?"
- Slogans you used to repeat have conditions attached.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What is the "always" or "never" in this claim, and what assumptions does it need to be true?
- ?Can I build one specific case — from an edge, from time, from failure, from concurrency — where it is false?
- ?If I found one, what narrower claim is true, and what mechanism makes it true?
- ?If I could not, what did I try, and where would the next person look?
What can go wrong
- Counterexamples are hunted for everything, including claims nobody depends on. "The product name is never empty" may have a counterexample; if nothing breaks when it does, the hunt was a distraction. Aim at the universals the design leans on.
- A counterexample is treated as a win rather than a requirement. Finding that a price change breaks the total is the start of a design decision, not the end of a review.
- The search is dishonest — three obvious cases, then "couldn't break it". The value of a failed search is proportional to how hard it tried, and a review that says "tried nothing, found nothing" should say so.
- Counterexamples from other systems are imported without checking that the assumption transfers. A race that breaks a shared-memory counter may be impossible here because the database serialises the write; the case has to be traced against *this* design.
- Counterexample hunting slows agreement. A claim that would have been accepted in a sentence now costs ten minutes and sometimes produces a design change nobody budgeted for.
- A found counterexample can be a case the product does not care about, and time is then spent deciding whether to care.
- The habit can make a reviewer tiresome if applied to every sentence; it is for the claims the design leans on.
- "A counterexample proves the design is wrong." It proves the *claim* is wrong as stated. Often the design is fine and the sentence needed narrowing; sometimes the case is real and unimportant. The counterexample starts a decision.
- "No counterexample means it is safe." It means nobody found one, with a known amount of effort. Tests and invariants are how the search is made repeatable; the counterexample is how the test was found.
- "This is adversarial." It is the opposite: a counterexample is the most respectful review possible, because it gives the author something concrete to answer. Vague doubt is what feels adversarial.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny claim with an "always" or "never" in it — about code, data, an API, a deployment, a process — can be attacked by constructing one case; the factories (assumptions, edges, time, failure, concurrency) are the same everywhere.
- ILLUSTRATIVEThe lamp at 40 and 45, the third-of-a-cent rounding and the two tabs are invented to show the move producing a case; the arithmetic is for the shape of the argument.
- TEAM-SPECIFICOn a team with a review culture the move is a norm and a reviewer is expected to bring a case; a solo learner has to play both roles, and the honest version is to write the claim down, leave it for a day, and attack it cold.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto's "review the LLM's answer" at /manifesto/review is counterexample thinking applied to generated code: every "this handles all cases" the model writes is a claim to attack with one case.