The Learning Signal
Every paradigm is defined by where the gradient's target comes from. That source decides what the model can be wrong about without anyone noticing.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
If you strip away the algorithms, what actually distinguishes supervised, unsupervised, semi-supervised and self-supervised — and what does that tell you to monitor?
A head of data science is reviewing four models before a platform migration: a churn classifier, a customer segmentation, a self-trained ticket router and a pretrained product encoder. They want one review checklist and are being told the four are too different to share one.
Review each model on its own metric: AUC for the classifier, silhouette for the segmentation, accuracy for the router, downstream accuracy for the encoder. Each number is the standard one for its paradigm, and all four look fine.
Each metric measures agreement with the target source. When the target source is the problem — a biased label process, an arbitrary distance, the model's own predictions, a duplicated corpus — the metric rewards agreeing with the problem.
- Each metric measures agreement with the target source. When the target source is the problem — a biased label process, an arbitrary distance, the model's own predictions, a duplicated corpus — the metric rewards agreeing with the problem.
- The four numbers are not comparable and are not meant to be, so the review has no way to say which model is at most risk. The one with the highest number may be the one with the least external check.
- The migration changes the feature pipeline for all four. Only the classifier has a label to notice; the other three will keep producing outputs of the same shape with no metric that can move.
- Nothing failed offline because the offline metric for three of the four is circular, and the fourth was computed on a selected population.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Each of the four has a different target source: an observed outcome, an objective over the inputs, a mix of observed and self-generated labels, and a target manufactured from the input. The review question is the same for all four: what is the gradient pointing at, and who checked that it points at the business?
- The decision downstream of the review is which models need what kind of monitor — because the source of the signal decides the shape of the invisible failure.
- One example, for review purposes, is one model with its target source, its loss, its evaluation set and the thing outside the loss that was used to check it.
- For the churn classifier the target is an observed cancellation; for the segmentation there is none; for the router it is eight thousand human tags plus the model's own predictions; for the encoder it is masked tokens.
- In every case something external — a label, a randomised campaign, a gold holdout, a downstream probe — either exists or does not, and that is the first row of the checklist.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Gradient-based learning moves parameters to reduce a loss. The loss compares the model's output to a target. The paradigm is the answer to "where does the target come from": observed in the world (supervised), computed from the inputs by an objective you chose (unsupervised), partly observed and partly generated by the model (semi-supervised), or manufactured from the input by hiding part of it (self-supervised).
- Each source has a characteristic blind spot. An observed label cannot be wrong about the label, only about whether the label means the outcome. A chosen objective cannot be wrong about the objective, only about whether the objective means anything. A self-generated label cannot disagree with the model. A manufactured label cannot know what the downstream task needs.
- Offline evaluation compares outputs to the same source, so it inherits the same blind spot. The only evaluation that can see the blind spot is one against something the loss never touched.
Four sources, one gradient
The loss is a function of the model's output and a target. The optimiser does not know or care where the target came from. That is the whole reason the four paradigms can share every algorithm and still fail in completely different ways.
Read each model by asking where the target comes from. The churn classifier's target was observed — a cancellation event. The segmentation's target is the objective — squared distance to a centre. The router's target is human tags plus its own predictions. The encoder's target is the tokens it hid.
Each source has a blind spot
The model can only be wrong relative to its target. Whatever the target cannot express, the model cannot be penalised for getting wrong, and the offline metric — computed against the same target — cannot show it.
That is a table, and it is the review checklist. The column that matters is the last one: what outside the loss would reveal the blind spot.
| Paradigm | Target source | Cannot be wrong about | Invisible failure | External check |
|---|---|---|---|---|
| Supervised | Observed label | The label | Label process ≠ outcome; selected population | Delayed outcome per segment, unselected holdout |
| Unsupervised | Chosen objective | The objective | Structure that means nothing to the business | Randomised outcome test per cluster |
| Semi-supervised | Labels + own predictions | Its own confidence | Confident errors become training data | Pseudo-label-free gold holdout from production |
| Self-supervised | Hidden input | The pretext | Corpus bias and dominant sources in the representation | Probes; downstream gold per slice |
The monitor follows from the source
Once the blind spot is named, the monitor writes itself. A supervised model needs a delayed-outcome check per population segment. A segmentation needs a re-validation on a schedule and a geometry-drift monitor. A self-trained model needs a gold set the loop cannot touch. An encoder needs probes and per-slice downstream metrics on every version.
The assumption all four share is the same: the target the gradient pointed at still corresponds to the thing the business acts on. It is just checked differently.
For each model, the target source that shaped its parameters still corresponds to the outcome the business acts on, and an external check the loss never saw confirms it.
holds when The external check has been run on the currently served population within its expiry, and the monitor matched to the paradigm's blind spot is live and unmuted.
breaks when The label process or population changes (supervised); the feature geometry shifts (unsupervised); the archive stops matching arrivals (semi-supervised); the corpus or encoder version changes (self-supervised).
respond Rerun the external check before touching the model. If it fails, the target source has moved, and retraining on the same source rebuilds the same blind spot.
How to build it
Most important first.
- For every model, write down the target source in one sentence and the external check in another. If the second sentence is empty, the model has no evaluation, whatever the metric says.
- Map the blind spot to a monitor: label-process drift and population drift for supervised; feature-geometry drift and periodic external re-validation for unsupervised; distribution match and a pseudo-label-free gold set for semi-supervised; probes and per-slice downstream metrics for self-supervised (Model Monitoring).
- Prefer the paradigm whose blind spot you can afford to check. A supervised model with a good label and a delayed-outcome monitor is easier to keep honest than a segmentation nobody will run a campaign for (Choosing a Paradigm).
- Treat the migration as a change to the inputs of all four, and check each one's assumption explicitly, because three of them cannot tell you (Train / Serve Skew).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For each model, the external number: realised churn per score band, campaign lift per segment, per-class routing accuracy on a refreshed gold set, downstream metrics per slice. These map to decisions. The paradigm-native metric does not.
- A simple per-model flag: does an external check exist, when was it last run, and does it cover the population being served. This is the review checklist.
- Do not rank the four by their native metrics. An AUC and a silhouette are not on a scale, and neither is a measurement of the business.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- For each model, the thing the gradient pointed at during training still corresponds to the outcome the business is acting on, as checked by an external measurement the loss never saw.
- The population each model is served on is the population its external check was run on, and a change in that population is detected by a monitor rather than assumed away.
- Every model's target source is documented precisely enough that a reviewer can name its blind spot without reading the code.
- Offline: for each model, locate the external check and confirm it is independent of the loss — no pseudo-labels in the holdout, no silhouette as evidence, no takedown-only counterfeit labels.
- Online: for each model, confirm a monitor exists that would move if the blind spot opened, and that it has fired at least once in a test.
- Over time: schedule the external checks with an expiry, and treat an expired check as a missing evaluation in the next review.
What can go wrong
- The checklist is filled in once for the review and never again; a year later the external checks have all expired and every model still reports its native metric.
- The monitor for the supervised model fires on population drift and is muted as noise; the unsupervised model has no monitor to mute.
- The encoder is treated as infrastructure and left out of the review, and the router and the segmentation both sit on it.
- A checklist that asks for an external check will declare some models unevaluated, which is unwelcome news about systems that have been in production for years.
- External checks are expensive in exactly the way the paradigm was chosen to avoid — labels, campaigns, expert review.
- Documenting the target source precisely forces the label-process conversation that most teams have been avoiding.
- "The four paradigms need four review processes." They need one question asked four times: what is the target, and what outside the loss confirms it means something.
- "Unsupervised models have no labels, so they cannot suffer label bias." They have a chosen objective instead, and the choice is a bias with no metric to reveal it.
- "Self-supervised models are more objective because the data labels itself." The data labels itself with whatever the pretext asks for, at the corpus's frequencies. That is less arbitrary than a human label and no more aligned with the business.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat offline evaluation compares outputs to the same target source the loss used, and therefore inherits its blind spot, follows from what a loss is and holds for every paradigm and model family.
- SIMPLIFIEDReinforcement learning, where the signal is a delayed reward from an environment, is left out; it has its own blind spot — the reward specification — and the same review question applies.
- CONTESTEDA serious position holds that the taxonomy is obsolete because modern systems mix all four — a self-supervised encoder, a supervised head, pseudo-labels for the long tail, and clustering for candidate generation — and reviewing by paradigm misses the interactions. The counter is that the mixing is exactly why the per-source question matters: each component has its own blind spot and the composite inherits all of them.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — an external check with an expiry date is a scheduled test whose absence should fail a review, which is a process discipline this domain assumes rather than teaches.