Semi-Supervised Learning
A few thousand labels and a few million unlabelled rows. The unlabelled data helps exactly when it comes from the same distribution as the labels — and that is the thing you cannot check with labels.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Labels are scarce and unlabelled data is abundant. Under what conditions does the unlabelled data make the model better rather than more confidently wrong?
A support team has hand-tagged eight thousand tickets with a reason code out of three million in the archive. They want every ticket routed to the right queue and cannot afford to tag more than a few hundred a week.
Train on the eight thousand, predict on the unlabelled three million, take the confident predictions as new labels, retrain on everything, repeat. The model bootstraps itself from a small seed and gets a big training set for free.
The seed labels came from a billing-incident fortnight, so the model is confident about billing and ignorant about everything else. Self-training amplifies exactly what it is confident about: billing pseudo-labels flood in, every other class starves.
- The seed labels came from a billing-incident fortnight, so the model is confident about billing and ignorant about everything else. Self-training amplifies exactly what it is confident about: billing pseudo-labels flood in, every other class starves.
- Confident wrong predictions become training labels and are never revisited. The model's errors are now in the training set with the same weight as human tags, and validation on the seed set does not contain them.
- The unlabelled archive includes discontinued products whose tickets look like nothing in production. Pseudo-labelling them teaches the model a distribution that no longer arrives.
- The offline metric on the held-out seed labels went up each round, because the pseudo-labels are the model's own predictions and agree with the model.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the reason code for a new ticket at creation time. The label is the code an experienced agent assigned; it is observed only for the tagged eight thousand.
- The decision is which queue the ticket enters, so the output is a probability over codes with a fallback to a human triage queue below a confidence cut (Thresholding).
- One example is one ticket: subject, body, product, customer tier, and — for the labelled subset — the reason code. The eight thousand were tagged from a two-week window last spring, when a billing incident dominated volume.
- The three million unlabelled tickets span four years, several products since discontinued, and two rewrites of the ticket form.
- Ongoing labels come from the routing outcome: whether an agent re-routed the ticket, which is a noisy, delayed proxy for the true code.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Semi-supervised methods add a term to the loss that uses unlabelled data: consistency (the prediction should not change under a small perturbation), entropy minimisation (predictions should be confident), or pseudo-labels (treat confident predictions as targets). All of them encode the assumption that decision boundaries should pass through low-density regions of the input.
- That assumption is the whole deal. If the classes really do form separated clusters in the input space, unlabelled points tell the model where the gaps are and a few labels tell it which cluster is which. If the classes overlap, or the unlabelled data comes from a different distribution, the same mechanism pushes the boundary somewhere confidently wrong.
- Nothing in the loss checks the assumption. The unlabelled term goes down whether the low-density regions are class boundaries or artefacts of how the archive was collected.
The loss term that assumes a gap
Every semi-supervised method adds a term the labels do not touch. Pseudo-labelling is the plainest: predict on unlabelled rows, keep the confident ones as targets, minimise the usual loss against them. Consistency methods require the prediction to survive a perturbation; entropy minimisation rewards confidence directly.
What all three share is the belief that a good decision boundary sits where the data is sparse. That is true when classes are clusters and false when they overlap, and the method cannot tell which case it is in.
1def pseudo_label_round(model, unlabelled, per_class_cap, min_conf):2 probs = model.predict_proba(unlabelled) # (n, k)3 conf, pred = probs.max(1), probs.argmax(1)4 keep = []5 for c in range(probs.shape[1]):6 idx = np.where((pred == c) & (conf >= min_conf))[0]7 idx = idx[np.argsort(-conf[idx])][:per_class_cap] # top-N per class, not "all above cut"8 keep.extend(idx)9 # these rows now carry the model's own prediction as their target10 return unlabelled[keep], pred[keep]11 12# the gold holdout is never passed through this function13for round in range(R):14 X_p, y_p = pseudo_label_round(model, X_unl, per_class_cap=500, min_conf=0.9)15 model = train(concat(X_gold_train, X_p), concat(y_gold_train, y_p))16 report_per_class(model, X_gold_holdout, y_gold_holdout) # any class regression -> stopThe cap is what stops one class from eating the rest. The last line is what stops the loop from reporting its own agreement as progress: the holdout has never seen a pseudo-label.
Where the unlabelled data came from
The three million tickets are not a bigger sample of the eight thousand. They are four years of history, discontinued products and a different ticket form, and the eight thousand are one fortnight dominated by one incident. The unlabelled term will happily find low-density gaps between last year's product and this year's, and call them class boundaries.
The cheapest test is to train a classifier to separate labelled from unlabelled rows. If it succeeds, the sets differ, and the unlabelled term should be restricted to rows it cannot separate.
looks like A training set that grew from eight thousand to four hundred thousand rows, each with a reason code, evaluated on a held-out set that also grew.
why it leaks If any pseudo-labelled row lands in the evaluation set, the metric measures agreement between the model and its own past predictions. The answer reached the test set through the model.
fix Freeze a gold holdout sampled from production before the first round; assert by row id that no pseudo-labelled row ever enters it; evaluate per class.
What must remain true after routing goes live
The model was trained on a mixture of gold and self-generated labels under the assumption that new tickets look like the unlabelled archive it was fed. That assumption is about the arriving distribution and has to be re-checked as products, forms and incident mixes change.
The delayed proxy — whether an agent re-routed — is the only production label, and it is noisy and can go quiet. It must be monitored as a signal in its own right.
New tickets are distributed like the unlabelled rows the model was self-trained on, and the gold holdout still represents current traffic.
holds when The labelled/unlabelled discriminator stays near chance on a weekly sample, and the monthly gold refresh shows per-class metrics within tolerance of the previous month.
breaks when A new product launches; the ticket form changes; an incident shifts the class mix so that one code dominates arrivals for a fortnight.
respond Spend the labelling budget on the new segment first, retrain with those rows as gold, and only then let pseudo-labelling near the new product.
How to build it
Most important first.
- Check the distribution match first: train a classifier to distinguish labelled from unlabelled rows. If it can, the unlabelled data is from somewhere else and the cluster assumption is already broken (Selection Bias).
- Hold out a labelled set that is sampled from the current production distribution, not from the seed window, and evaluate every round on it. Never let a pseudo-label into that set.
- Balance pseudo-labels per class and cap the confidence threshold's effect: take the top-N per class per round, not everything above a cut, so a confident class cannot starve the rest.
- Spend the weekly labelling budget on the rows the model is least certain about, or where the labelled/unlabelled classifier says coverage is thin, and treat those as gold rather than pseudo (Label Quality).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Macro-averaged recall per reason code on the production-sampled holdout. That is the number the routing decision depends on; it exposes a class that self-training starved.
- The re-route rate per predicted code in production — the delayed proxy label — as a check that confident predictions are being acted on correctly.
- Accuracy on the seed-window holdout goes up every round and is the one number not to trust: it measures agreement with a distribution the model was fed its own predictions from.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The unlabelled data arriving in production comes from the same distribution as the labelled seed, or the mismatch has been measured and the unlabelled term is restricted to rows the labelled/unlabelled classifier cannot tell apart.
- The low-density boundary assumption holds for these classes: reason codes form separable regions in the representation, which a per-class calibration check on the gold holdout can test.
- Pseudo-labels never enter the evaluation set, and the gold holdout is refreshed from current traffic on a schedule.
- Offline: labelled-versus-unlabelled discriminator AUC as a distribution-match test; per-class metrics on a gold holdout after every self-training round, with a rule that any class regression stops the round.
- Online: shadow the routing for a week and compare predicted code against the agent's final code, per class, before any queue is switched.
- Over time: refresh the gold holdout monthly with a sample from live traffic, tagged by hand, and re-run the round-by-round evaluation on it.
What can go wrong
- The per-class cap keeps the classes balanced and the pseudo-labels are still wrong within a class, because the confident errors are systematic — every "refund" ticket mentioning a card gets labelled "billing".
- The production-sampled holdout is collected once and ages; a year later it no longer represents traffic and the model is validated against last year.
- The re-route signal is used as a label and agents stop re-routing because the queue is too busy, so the proxy label goes quiet exactly when the model is worst.
- A production-sampled gold holdout costs the same scarce labelling budget the method was supposed to save, and it is the only thing that makes the method safe.
- Per-class caps and uncertainty-targeted labelling make each round smaller and slower than "take everything above the cut".
- Restricting the unlabelled term to distribution-matched rows may throw away most of the three million, which was the whole attraction.
- "We have three million rows, so the model will be well trained." Three million rows without labels teach the model where the data is, not what it is. They help only where the cluster assumption holds.
- "Held-out accuracy went up every round, so self-training worked." The holdout was from the seed window and the pseudo-labels are the model's own predictions; the metric measures self-agreement.
- "Confident predictions are reliable enough to use as labels." Confidence is a property of the model, not of the truth. A model that has only seen billing tickets is confident about everything being billing.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat every semi-supervised objective encodes a low-density-boundary or cluster assumption, and that pseudo-labels reinforce the model's existing errors, holds regardless of which method or model family is used.
- DATA-SPECIFICConsistency-based methods work well on images, where perturbations like crops and colour shifts are label-preserving by construction; on tabular or text data the perturbation is harder to define and pseudo-labelling with strict per-class caps is usually the safer variant.
- CONTESTEDA serious position holds that with a pretrained encoder available, semi-supervised learning is obsolete: fine-tune the encoder on the few thousand labels and skip the pseudo-label loop, because the representation already encodes the structure the unlabelled data would have provided (Self-Supervised Learning, Transfer Learning). The counter is that domain-specific archives — internal ticket formats, product names — are exactly what a public pretrained encoder has not seen, and a small self-training loop on top of it recovers that at low cost.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — asserting by row id that no pseudo-label ever enters the evaluation set is a test fixture discipline; the refresh schedule for the gold holdout is a maintenance question this domain assumes.