Learning vs Programming
A program encodes rules someone wrote. A model encodes patterns from data it was shown — and therefore inherits the data's biases, gaps and timing. The model is a set of assumptions with weights attached.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What is actually different about a component whose behaviour was learned from data rather than written as rules, and what does that difference do to testing, review and change?
A backend team is replacing a hand-written fraud rule set with a learned model. Their review process, test suite and rollback procedure were built for code. They want to know what carries over and what does not.
A model is a function like any other. Treat it as a library: write tests for the outputs, review the code that calls it, and deploy it with the same pipeline.
The unit tests pass — for the inputs the team thought to write. The model's behaviour on the inputs nobody wrote a test for is fixed by data nobody reviewed, and the fraud ring the training data under-represents sails through.
- The unit tests pass — for the inputs the team thought to write. The model's behaviour on the inputs nobody wrote a test for is fixed by data nobody reviewed, and the fraud ring the training data under-represents sails through.
- A code review of the model reads the training script, which is fifty lines and says nothing about what was learned. The reviewer approves a function whose behaviour they have not seen.
- The rollback procedure restores the previous artifact but not the previous feature pipeline, and the "previous" model now runs on inputs it never saw.
- The rules were wrong in ways the analysts could name. The model is wrong in ways that are a property of the data's gaps, and nobody can name them until production does.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The rules and the model both output "block or allow" for a transaction. The rules encode what analysts wrote down; the model encodes what the historical chargeback data implied.
- The lesson's target is the difference in *where correctness lives*: in the text of the rules, or in the relationship between the training data and the present.
- The rule set is a few hundred lines with explicit thresholds — amount, velocity, country mismatch — each with a comment explaining the analyst's reasoning.
- The model was trained on two years of transactions with chargeback labels. Fraud patterns from the second year dominate; a fraud ring that operated in the first year and was shut down is under-represented.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A program's behaviour is specified by its text; its correctness can be checked against that text and reasoned about locally. A model's behaviour is specified by the training data, the objective and the function family; the training script is a recipe, and the learned weights are what the recipe produced from that data.
- Because the weights come from the data, every property of the data becomes a property of the model: what the data over-represents the model is confident about; what it under-represents the model extrapolates on; what changed after the data was collected the model does not know. The model inherits the data's biases, gaps and timing exactly.
- The model is therefore a set of assumptions with weights attached: that the population is the training population, that the relationship between features and outcome is the training relationship, that the features mean what they meant. The weights are the only part that is written down.
Where the behaviour is written
Put the two side by side. The rule says: block if amount is above a threshold and the country does not match the card's. Its behaviour on every input follows from its text. The model says: block if a weighted combination of forty features crosses a threshold, where the weights were chosen to minimise a loss on two years of labelled transactions. Its behaviour on every input follows from those two years.
This is not a difference in accuracy or in complexity. It is a difference in where the specification lives, and therefore in what a review, a test and a rollback have to cover.
The training script: data loading, a model class, a fit call, a metric. Fifty lines, all reasonable, none of which determines what the model does.
What one example is, how labels were made, what period and population the data covers, what is under-represented, what the objective optimised, and slice-level results on the known gaps.
The learned weights are a function of the data; a review that does not cover the data has not reviewed the behaviour. The training script would produce a different model from different data with no change to its text.
Inheriting the data
The fraud model inherited three things from its training set that no line of code expresses. Timing: two years ending at a date, after which it knows nothing. Gaps: a fraud ring shut down in year one appears in a handful of examples, so the model has almost no opinion about that pattern. Bias: the labels are chargebacks, so transactions the old rules blocked have no label at all, and the model learned fraud only as the old rules failed to catch it.
None of these is a mistake in the training. They are properties of the data faithfully transferred to the model, and each is an assumption that will be tested by production without anyone asking it to be.
The transactions being scored come from the same population, in the same proportions, as the two years the model learned from.
holds when The merchant mix, the geographic mix, the fraud patterns and the share of new cardholders are stable relative to the training window.
breaks when A new market launches; a fraud ring adapts to the model; the old rule set is switched off and transactions it used to block now reach the model with no training analogue.
respond Check whether quality moved on the shifted slices before doing anything; if it did, retrain with a window and sampling that covers the new population, and re-run the slice evaluation.
Assumptions with weights attached
The slogan is worth making precise. A trained logistic regression is a vector of weights and a bias. Applying it to an input assumes that the input's features are on the scale the weights were learned for, that the linear relationship the weights express still holds, and that the intercept still reflects the base rate. Remove any assumption and the same weights produce a wrong answer with the same confidence.
The code below is the entire model. Everything it assumes is in the comments, and none of it is in the code.
1import numpy as np2 3# w and b were learned on transactions from 2024-01 to 2025-12.4w = np.array([0.8, -0.3, 1.9, 0.05]) # amount_z, tenure_z, country_mismatch, hour5b = -2.1 # encodes the base fraud rate of that period6 7def score(x):8 # assumes: x is standardised with the *training* mean and std,9 # country_mismatch is 0/1 with the same country table,10 # hour is local time as the training pipeline computed it,11 # the base rate is still roughly what b encodes.12 z = w @ x + b13 return 1.0 / (1.0 + np.exp(-z)) # P(chargeback) under those assumptionsChange the standardisation constants, the country table, the time zone or the base rate and the four numbers in w are still "correct" — for a world that no longer exists. The rule set would have needed an edit; the model needs a monitor.
How to build it
Most important first.
- Review the data and the objective, not just the training script: what is one example, what is the label rule, what is under-represented, what period does it cover (Dataset Construction, Target Definition).
- Replace "unit tests of outputs" with the ML test stack: invariants the model must satisfy on whole classes of input, slice-level evaluation, and a serving contract (Model Invariant Tests, Evaluation Slices).
- Version the artifact with the data, label rule and feature code it was learned from, so a rollback restores the whole set of assumptions and not just the weights (Model Lineage).
- Write the assumptions down as the model's specification, and monitor each one; this is the only form of review that says anything about behaviour on inputs nobody tested (Don't Delegate Understanding).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Slice-level quality on the populations the training data under-represents — the number that tells you where the learned assumptions are thinnest. Aggregate quality hides exactly this.
- For each named assumption, a monitor: population distribution, feature meaning, outcome relationship. These are the model's "tests" in the sense that code tests are code's.
- Do not measure a model by "the tests pass". The tests cover the inputs someone imagined; the model's behaviour is defined on all of them.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The population of transactions being scored is still the population the training data was drawn from, in the proportions that matter for the decision.
- The relationship the model learned between the features and chargebacks still holds — fraudsters have not adapted to the model's own behaviour.
- Each feature still means what it meant when the weights were learned: same definition, same unit, same source.
- The under-represented regions of the training data are still rare in production, or the model's behaviour there has been checked explicitly.
- Offline: evaluate on slices chosen from the data's known gaps — the old fraud ring's pattern, new countries, high-amount transactions — rather than on the aggregate.
- Online: monitor the feature distributions and the flagged rate per slice; a slice moving out of the training distribution is an assumption breaking before quality can show it.
- Over time: as chargebacks arrive, compare precision per slice with the offline slice evaluation; a slice that decays first is where the learned assumptions were thinnest.
What can go wrong
- Invariant tests are written for the properties the team can articulate, and the model's actual failure is on a property nobody thought to require — monotonicity in amount, say, that fraud data does not support.
- The assumptions are written down at launch and never revised; a year later the model is running on a population the assumptions no longer describe and nobody re-reads them.
- A learned component is put behind a rule-based guardrail, and the guardrail becomes the real decision-maker while the model's quality quietly decays unmeasured.
- The rule set was reviewable line by line and the model is not; the team trades local reasoning for a function that captures patterns no analyst wrote down, and pays for it in a monitoring discipline that rules never needed.
- Slice evaluation and invariant tests require the team to name the slices and invariants, which is domain work the model was supposed to save.
- Keeping the rules as a guardrail preserves the reviewability but caps the model's value at what the rules allow.
- "A model is just a function; test it like one." Its behaviour on untested inputs is fixed by data, not by code, and a passing test suite says nothing about the inputs it does not contain.
- "The training code was reviewed, so the model was reviewed." The training code is the recipe. Reviewing it without the data is approving a dish from the ingredient list.
- "The model learned the rules the analysts wrote, plus more." It learned whatever the labelled data implied, which includes the analysts' past decisions where those shaped the labels, and excludes anything the data did not contain.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model's behaviour is a property of its training data holds for every learned component from a logistic regression to a foundation model; the foundation model differs only in that the data was somebody else's.
- CONTESTEDA serious position holds that the distinction is overdrawn: hand-written rules also encode assumptions about a past population, drift just as badly, and are reviewed far less carefully than their text suggests. That is right as far as it goes; the difference that survives it is that a rule's assumptions are written down where a reviewer can read them and a model's are not.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — what it means to test a component whose behaviour is defined on all inputs by data rather than on some inputs by code is a testing-theory question; this domain supplies the ML test stack and assumes the general theory.