Characterization Tests
Tests that record what the code does today — including what it does wrongly — so that a later change has a baseline to be measured against. They assert behaviour, not correctness.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
How do I get a safety net around code whose correct behaviour nobody can state?
Finance needs a new discount tier added to order pricing. The pricing code has no tests, and when asked what it currently does with stacked discounts, three people give three answers.
Write tests for what the pricing code is supposed to do. Read the code, work out the intended behaviour, and assert that. Where the code is wrong, the test will fail, and now you have found bugs as a bonus.
The suite goes red immediately, on behaviour that customers have relied on for four years, and the "bonus bugs" turn out to be the product. Half a day disappears into arguing which failures are real.
- The suite goes red immediately, on behaviour that customers have relied on for four years, and the "bonus bugs" turn out to be the product. Half a day disappears into arguing which failures are real.
- Worse, the tests get quietly adjusted to pass, which means they now assert whatever the code does anyway — but with a comment claiming they assert intent, so the next reader trusts them for the wrong reason.
- When the discount tier is finally added and three tests fail, nobody can tell whether those failures are the intended new behaviour or a regression, because the suite never distinguished the two.
- As requirements keep arriving, this compounds: an intent-asserting suite over unspecified code becomes a second source of truth that disagrees with the first, and reconciling them is a project of its own.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The change is due in two weeks, so the net has to be built in days, not months.
- There is no specification. The only authority on current behaviour is the running system and its data.
- Production data cannot be copied into the test environment unmasked, so inputs have to be synthesised or scrubbed.
- The code is deeply entangled with the database, so any test that requires a clean unit boundary requires a refactor first — which is the thing we do not yet dare to do.
- A characterization test asserts what the system does, never what someone believes it should do. The moment it asserts intent, it stops being a baseline.
- Writing the net must not change behaviour. If a test forces a code change to pass, the code change is the risk you were trying to avoid.
- Every quirk the suite pins must be identifiable later as a quirk, so that a deliberate behaviour change is distinguishable from a regression.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The suite owns exactly one claim: *this is what the system did on the day we wrote it.* Nothing about correctness.
- A separate artefact — a list, a ticket, a comment on the test — owns the observation that a pinned behaviour looks wrong. That is a business conversation, not a test failure.
- The person making the upcoming change owns deciding which pinned behaviours are allowed to move, before making it.
- Write the tests at the coarsest boundary that is reachable without changing code — usually a public entry point, sometimes an HTTP endpoint, occasionally the database state after a job runs. Coarse is a feature here: it pins behaviour without pinning structure (Where a Test Must Be Real).
- This is the opposite of the usual advice to test small units, and deliberately so: a fine-grained suite over code you are about to restructure will break for reasons that have nothing to do with behaviour (What a Unit Is).
- The boundary should be one you expect to keep. If the entry point itself is going to move, pin one level out from it.
Assert what it does, not what it should do
The whole discipline is in one substitution: run the code, look at the answer, and write the answer down. Not the answer you expected — the answer you got.
The difference sounds trivial and is not, because the intent-asserting version fails on day one and gets negotiated into either a code change or a weakened test. Both outcomes destroy the baseline you were trying to establish.
// stacked discounts "should" add: 10% + 5% = 15%
test('stacked discounts add', () => {
expect(price(100, ['SAVE10', 'EXTRA5'])).toBe(85)
})
// RED. Actual is 85.50 — they compound.
// Now: change the code, or change the test?// NOTE: discounts compound (100 * 0.9 * 0.95), they do not add.
// Unspecified; four years of invoices depend on it. Pinned, not endorsed.
test('stacked discounts compound — see PRICING-412', () => {
expect(price(100, ['SAVE10', 'EXTRA5'])).toBe(85.5)
})The second test is green on the day it is written, which means it can protect the refactor that starts tomorrow. The first test is a bug report wearing a test's clothes — and bug reports do not catch regressions. Splitting "is it safe to change" from "should it change" is what lets the two-week deadline be met at all.
Building the net in a day
The order matters, because the first step is the one people skip: freeze non-determinism before capturing anything. A suite that records Date.now() looks like it works and rots overnight.
Everything here is deliberately cheap. Characterization is scaffolding, and scaffolding that takes three weeks to erect has failed regardless of how well built it is.
- 1Find the reachable boundary
Pick the coarsest entry point you can call without editing code — a function, a handler, a job.
fails by Choosing a boundary that requires a refactor to reach, which is the risk you were avoiding.
- 2Freeze the world
Inject or stub clock, ids, random and locale so the same input gives the same output twice.
fails by Skipping it, then spending two days debugging a suite that fails only after midnight UTC (Time as a Dependency).
- 3Get inputs from reality
Replay scrubbed production requests, or sweep the parameter space, rather than inventing cases.
fails by Handwritten inputs that encode exactly the assumptions you are trying to test.
- 4Capture and read the output
Snapshot the full observable result. Read it. Note anything that surprises you.
fails by Approving a snapshot without reading it, which pins bugs invisibly instead of visibly.
- 5Label the surprises
Name the test after the quirk and link a ticket, so the pin is distinguishable from an endorsement.
fails by An unlabelled pin, which the next engineer reads as intended behaviour.
- 6Prove the net catches things
Break the code on purpose. Confirm red. Revert.
fails by Trusting a suite that has never failed — the most common way a characterization net turns out to have been asserting nothing.
Steps two and six are the ones that get dropped under time pressure, and they are the two that decide whether the suite is real (Refactoring Without Tests).
What goes wrong with the net itself
A characterization suite is a mitigation, and mitigations have their own failure modes. Most of them come from forgetting that the suite is temporary and structurally coupled to the code it covers.
The recurring theme: a suite that nobody reads, nobody trusts, or nobody can run quickly has all the maintenance cost of a real test suite and none of the protection.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Snapshot grows past a few hundred lines | Diffs are approved without reading; a real regression sails through | The captured surface includes everything rather than the observable contract | Capture a projection — the fields that constitute the behaviour — and let the rest vary |
| Test uses the real clock or real ids | Flaky failures near midnight, at month end, or on reruns | Non-determinism was never pushed to a seam | Inject time and id generation before writing any assertions (A Deterministic Core) |
| Pinned quirk is unlabelled | Someone "fixes" a failing test and reintroduces a closed incident | The suite recorded behaviour but not the fact that the behaviour was unexamined | Name every deliberate pin after the quirk and link the ticket |
| Suite takes eleven minutes | It is run once per PR, never during refactoring | Boundary chosen at the database or HTTP layer for convenience | Keep a fast subset covering the change's blast radius; run the full suite in CI (The Refactoring Loop) |
| Behaviour is deliberately changed | Forty tests red, and a scramble to decide which are expected | Correct and unavoidable — the suite is doing its job | Review them as a batch, as a design activity, before merging; that review is the point of having had the suite |
How to build it
Most important first.
- Pick the paths the upcoming change will touch. A characterization suite over the whole module is a nice idea that finishes after the deadline; the suite you need covers the blast radius of one change (Change Amplification).
- Generate inputs rather than inventing them: replay scrubbed production requests, or sweep parameter combinations, because handwritten inputs encode your assumptions and your assumptions are the thing under suspicion.
- Capture the output, look at it, and assert it verbatim — an approval or snapshot test is the honest shape of this. If the captured output is surprising, that is information, not a reason to fix it yet.
- Label the surprises. A test named
stacked_discounts_apply_multiplicatively_which_is_probably_a_bugcosts nothing and tells the next reader exactly what they are looking at. - Freeze anything non-deterministic at the boundary — clock, ids, random, locale — or the suite records noise and fails on Tuesdays (Time as a Dependency, Randomness as a Dependency).
- Delete the suite when it has done its job, if the code it covered has since acquired real behaviour tests. Characterization tests are scaffolding with a legitimate end date.
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Before: the discount tier costs two weeks, most of which is reading code and manually re-checking twenty scenarios after each edit, with no accumulation — the next change repeats the whole thing.
- After: the discount tier costs a day of edits plus a suite run, and — this is the actual return — the change *after* that one costs the same day, because the net persists.
- The suite has its own carrying cost: it must be run, kept fast, and eventually reviewed test by test when the pinned behaviour is deliberately changed. Budget a day for that review at the moment the business asks to fix a quirk.
- What does not get cheaper: any change that alters the shape of the boundary you pinned. Coarse tests are cheap to write and expensive to move, and moving them is the price of the coarseness.
- You are deliberately pinning behaviour you believe to be wrong, which feels bad and is occasionally the wrong call — for a security defect or a data-corruption bug, pin it, then fix it immediately rather than preserving it.
- Coarse tests are slow and give poor failure localisation. A red snapshot tells you something moved, not what, and bisecting that is real work.
- It buys safety without buying understanding. A team can characterize a module, change it safely, and still not know what it does — which is fine for one change and corrosive over five years.
What can go wrong
- The suite pins a quirk that a customer contract actually requires, nobody labels it, and two years later someone "fixes" the failing test and reintroduces the incident it was documenting.
- Snapshots grow to thousands of lines and stop being read. A diff nobody reads approves anything, and the net is now decorative.
- Non-determinism leaks in and the suite becomes flaky, at which point people rerun it until it passes, which is the same as not having it (Refactoring Without Tests).
- The mitigation fails in its own way: a suite so slow it is only run in CI means the tight refactor loop it was meant to enable never happens.
- The suite depends on the current code, tightly and intentionally. That coupling is the point, and it is why it is temporary.
- It depends on determinism, which usually has to be introduced first — the smallest possible seam, injected before anything else moves (Seams).
- It does not depend on a specification, which is exactly why it is available when a specification is not.
- "So characterization tests replace unit tests." They are a different instrument for a different situation. Once behaviour is understood and specified, write tests that assert intent; keeping the snapshot suite as your only tests means the code can never be deliberately corrected without a mass review.
- "So we should snapshot everything." Snapshots over already-understood code pin implementation detail and make every legitimate change noisy. The technique is for unspecified behaviour.
- "Recording the bug means we approve of it." It means you have separated two decisions — *can we change this safely* and *should this behaviour change* — and are taking them one at a time. That separation is the entire technique.
- "We do not need these because we have monitoring." Monitoring tells you afterwards, in production, on customer traffic. That is a different risk posture, and for a pricing change it is the wrong one (Designing for Failure).
Testing it, and how it ages
- Verify the net before trusting it: deliberately break the code — flip a comparison, change a rounding mode — and confirm the suite goes red. A net that has never caught anything is unproven.
- Assert the whole observable output, not one field. Characterization is about catching what you did not think to check.
- Keep the suite runnable in under a minute on the paths being changed, because its value is in a tight edit-run loop, not in CI (The Refactoring Loop).
- As real behaviour tests are written — tests that assert what the business intends — the characterization suite should shrink. If it never shrinks, nobody ever established intent, and the module is still legacy with a nicer safety net.
- The suite becomes wrong the moment behaviour is deliberately changed, and that is correct: updating it is how a deliberate change gets recorded.
- Long-lived characterization suites tend to calcify into de-facto specifications, which is a mixed outcome — better than nothing, worse than someone writing down what the rules are (Documentation Decay).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThe technique only assumes you can run the code and observe an output, so it applies from COBOL batch jobs to serverless handlers; what changes is the boundary you can reach, not the idea.
- LANGUAGE-SPECIFICIn a language with cheap value equality and structural serialisation — Python, Ruby, TypeScript — snapshotting an output is a two-line affair. In C++ or Java with entity objects and identity equality, capturing a comparable snapshot needs deliberate serialisation work, so the technique costs meaningfully more up front and teams reach for it less often.
- CONTESTEDThe strongest opposing view: pinning behaviour you believe is wrong is preserving bugs and can entrench them for years, because a passing test reads as an endorsement to everyone who arrives later. Practitioners who have inherited suites full of unlabelled quirks argue you should establish intended behaviour first, fix, and test the fix — accepting the risk. That is right when the quirk is a defect with a known victim, and wrong when the quirk *is* the product and you cannot yet tell the difference.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — approval testing, snapshot management and mutation testing as ways to establish that a suite would actually fail; the mechanics live there, the design decision lives here.