Supervised Learning
A labelled target turns learning into function fitting. The model is only as right as the label, and the label was made by a process nobody wrote down.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
When someone hands you a table with a target column, what exactly has the model been told to learn — and who decided it?
A lending team wants to know which loan applications will default. They have four years of approved loans with a defaulted flag and would like a score they can put in front of underwriters.
A target column exists, so this is supervised learning: fit a classifier to defaulted, check the validation metric, and hand underwriters the score. Supervised learning is the well-understood case — a clean X and y.
The model scores applicants the old policy would have declined, and it has never seen one with an outcome. Its confident scores on that population are extrapolation dressed as prediction.
- The model scores applicants the old policy would have declined, and it has never seen one with an outcome. Its confident scores on that population are extrapolation dressed as prediction.
- The label definition changed twice. The model learned a blend of three targets and reports one validation number for all of them.
- Underwriters start following the score, so the approved population shifts toward what the model likes. Next year's training set is the model's own choices with outcomes attached (Feedback Loops).
- None of this shows up offline. The split was random, the label column was present, and the metric was computed on the same biased population the model was trained on.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an approved loan will be 90 days delinquent within 24 months. The label is a boolean derived from the servicing ledger; it exists only for loans that were approved and reached 24 months of age.
- The downstream decision is approve, decline or refer, so the output is a probability that an underwriting policy turns into one of three actions (Prediction vs Decision).
- One example is one approved application joined to its outcome: applicant fields at application time, bureau attributes pulled the same day, and the label from the ledger two years later.
- The rows come only from applications the previous policy approved. Declined applicants have no outcome, so the dataset says nothing about them (Selection Bias).
- The
defaultedflag was computed by an analyst's SQL query that changed twice over the four years as the definition of delinquency moved.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Supervised learning minimises a loss between the model's output and a provided target:
L(f(x), y). The gradient of that loss is the learning signal, and it points whereverysays it should. The model has no other source of truth about the world. - This makes the label a specification. Every choice in producing it — the window, the threshold, which rows have one at all — becomes part of what the model learns, silently and with equal weight to the actual pattern.
- The offline metric compares
f(x)to the sameyon held-out rows. It therefore measures agreement with the label process, not with reality. When the label process is biased, the metric rewards reproducing the bias.
The label is a specification
A supervised model minimises a loss against y. Nothing else enters the gradient. So whatever produced y — a SQL query, an annotator, a business rule, a survey — is the thing being learned, and every quirk of that process is indistinguishable from signal.
For the lending team, defaulted was three different queries over four years. The model does not know that. It sees one column and finds the function that best predicts it, which is a weighted blend of three functions, with weights set by how many rows each period contributed.
Every cell here is an approved loan, because only approved loans have a label. The matrix cannot contain the declined applicants the model will be asked about — that column is missing from the world, not from the table.
1-- label_v3: 90+ days delinquent within 24 months of origination2SELECT l.loan_id,3 l.originated_at,4 MAX(CASE WHEN d.days_past_due >= 905 AND d.as_of < l.originated_at + INTERVAL '24 months'6 THEN 1 ELSE 0 END) AS defaulted,7 'v3' AS label_version8FROM loans l9LEFT JOIN delinquency_snapshots d ON d.loan_id = l.loan_id10WHERE l.originated_at < CURRENT_DATE - INTERVAL '24 months' -- label must be mature11GROUP BY l.loan_id, l.originated_at;The WHERE clause is the part people forget. A loan originated last month cannot be labelled negative; it is unlabelled. Including it teaches the model that recent means safe.
Who has a label
The dataset contains only approved loans, because only approved loans have outcomes. That is not a flaw to fix with cleaning; it is the shape of the world the previous policy created. The model will be asked about applicants that policy declined and has no evidence about them.
The offline metric cannot see this. Held-out rows come from the same selected population, so the model is evaluated exactly where it is strongest and never where it will extrapolate.
Strong discrimination on a random held-out set of approved loans; calibration curve close to the diagonal on the same set.
Underwriters report the score is confident and wrong on thin-file applicants; realised default in the lowest-risk score band is above the predicted rate for that segment.
- 1Thin-file applicants were mostly declined by the previous policy, so the model has almost no labelled examples of them and extrapolates from the nearest approved ones.
- 2The label definition changed mid-window; the most recent version, which the business now uses, was the smallest share of the training set.
- 3Loans originated late in the window had immature labels and were counted as non-default.
What must remain true
The model encodes a contract with the label: this definition, this window, this population. Deploying it promises that the outcome the business cares about still matches the one that was learned, and nothing in the artifact enforces the promise.
So the assumption has to be checked as labels mature, and the population being scored has to be compared against the population that was labelled, from the first day.
The versioned label the model was trained on is the outcome the business is acting on, and the scored population is inside the labelled one.
holds when The label definition is frozen in code with a version stamped on every training row, and the extrapolation share on scored traffic stays low.
breaks when The delinquency definition changes for regulatory reasons; a marketing push brings a new applicant segment; the model itself reshapes who gets approved and therefore who gets labelled next.
respond If the definition moved, it is a new task: relabel and retrain against the new definition. If the population moved, restrict or hold out before retraining — retraining on the model's own approvals deepens the loop.
How to build it
Most important first.
- Write the label down as code with a version, and treat a change to it as a new task with a new model (Label Construction, Target Definition).
- Audit who has a label and who does not. If the labelled population was selected by a previous decision, say so in the model card and constrain the model's use to that population, or get outcomes for a sample of the excluded population (Selection Bias).
- Choose the split to match the deployment: a temporal split for a model that predicts the future, a grouped split when one applicant appears many times (Choosing a Split Strategy).
- Hold out a slice of applications that the model does not decide, so the future training set contains outcomes the model did not select (Exploration vs Exploitation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric that maps to the decision is the default rate among approved loans at the operating threshold, on a period after the training window — not validation AUC on a random split.
- Track the share of scored applications that fall outside the labelled population's feature range. That number says how much of production is extrapolation.
- Label-definition version per training row. If a model was trained across two versions, the metric is an average over two tasks.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The label computed at training time means the same thing as the outcome the business cares about at serving time, and its definition has not changed since.
- The population being scored is the population that had labels, or the model's behaviour outside that population has been checked separately.
- The features available at scoring time are the ones present at the moment the training rows were labelled, not refreshed afterwards.
- Offline: recompute the label from raw ledger events with the versioned definition and diff it against the training column. Any disagreement is a second target hiding in the first.
- Online: compare the feature distribution of scored applications against the labelled population, and measure the extrapolation share on day one.
- Over time: as outcomes mature, compare realised default rates against predicted probabilities per score band, including the held-out unselected slice.
What can go wrong
- The label window is shorter than the outcome takes to happen, so recent rows are mostly labelled negative and the model learns that recency is safety (Ground-Truth Delay).
- A feature is computed after the label event — the bureau pull is refreshed at charge-off — and the model reads the answer (Label Leakage).
- The held-out slice for unbiased outcomes is removed to save money, and two years later the training set contains only applicants the model approved.
- A held-out slice that the model does not decide costs real defaults on purpose; it is the price of a training set that is not the model's own echo.
- Versioning the label and refusing to mix definitions means fewer training rows per model and a smaller validation set.
- Constraining the model to the labelled population means the score is unavailable exactly where underwriters most want help.
- "There is a label, so the problem is well-posed." The label is a specification written by someone, under a policy, with a window. Well-posed means you have read it.
- "Validation accuracy is high, so the model understands default." It reproduces the label process on rows the label process selected. Whether that is default is a separate question.
- "More labelled data will fix it." More rows from the same selected population make the bias more precise, not smaller.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the loss gradient points wherever the label says follows from the definition of supervised learning and holds for every model family, from a linear model to a transformer.
- DOMAIN-SPECIFICSelection by a previous decision is severe in lending, hiring and fraud, where the label exists only for cases someone chose to act on; in image classification the labelled population is usually the whole population and the concern shifts to annotator agreement.
- CONTESTEDA serious position holds that reject inference — modelling the outcomes of declined applicants — is a statistical fiction, and that the honest answer is to state the population and stop. The other side answers that the model will be used on declined-like applicants whatever the documentation says, so an imperfect estimate beats a confident extrapolation.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a versioned label definition with a recomputation test is a contract test on the target itself; the discipline of keeping the recomputed label in agreement with the stored one is a testing question this domain assumes.