Label Quality
The label is the thing the model is trained to reproduce. Noisy, delayed, drifting, disputed or machine-generated labels put a ceiling on everything downstream, and the ceiling is invisible in the metric.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Where did the labels come from, how wrong are they, when did they arrive, and has what they mean changed since the first row was labelled?
A support platform wants to route incoming tickets to the right team automatically. Historical tickets carry a team label that agents set when they resolved them, and the model trained on those labels routes noticeably worse than the agents say it should.
The closed ticket says which team handled it. That is the ground truth; train on it and the model will learn the routing the agents do.
The label is a mix of "who should handle this", "who happened to close it", and "what the old router guessed and nobody bothered to fix". The model learns the mixture, and its ceiling is the agreement rate between those three things, not the true routing.
- The label is a mix of "who should handle this", "who happened to close it", and "what the old router guessed and nobody bothered to fix". The model learns the mixture, and its ceiling is the agreement rate between those three things, not the true routing.
- Two teams were merged eighteen months ago, and tickets before the merge carry labels for teams that do not exist. The model learns a team that cannot be routed to, or the labels were bulk-remapped and now describe a structure that never handled those tickets.
- The old router's guesses are in the labels, so the new model learns the old router's mistakes as truth and reproduces them with a higher validation score than the old router ever had.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the team that should handle a new ticket. The label is the team recorded on the resolved ticket, which was set by whoever closed it, sometimes after two reassignments.
- The decision is a single routing choice at ticket creation, and a wrong route costs a reassignment delay measured in hours.
- One example is one ticket at creation time — subject, body, customer segment — with the team on the closed ticket as the label. Three years of tickets; the team structure was reorganised twice in that period.
- A quarter of tickets were reassigned at least once; the recorded team is the last one, which may be the one that closed it as a courtesy rather than the one that should have handled it.
- For the last eighteen months, an earlier rule-based router set the initial team, and agents changed it only when it was badly wrong.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Supervised learning minimises disagreement with the labels. If labels are wrong at some rate, the best achievable model disagrees with the *true* answer at least as often as the labels do, and the offline metric — which compares to the same labels — cannot see the difference between a model that learned the truth and one that learned the noise.
- Label problems come in distinct kinds with distinct mechanisms. Noise: the label is randomly or systematically wrong. Delay: the label arrives late, so recent rows are unlabelled or mislabelled as negative. Definition drift: what the label means changed over the dataset's span. Disagreement: reasonable annotators would not agree, so there is no single true label. Model-generated: the label was set by a previous model and carries its errors (Feedback Loops).
- Systematic noise is worse than random noise. Random noise lowers the ceiling; systematic noise — the old router's pattern, the courtesy-close pattern — is learnable, and the model learns it as if it were signal.
Five kinds of bad label
These are different mechanisms with different symptoms and different fixes, and lumping them as "noisy labels" hides the one that is actually present. The table gives each its signature, which is usually visible in the label prevalence over time or in a blind relabelling before anything is visible in the metric.
The last row is the one that compounds: once the deployed model produces the labels, the training signal is the model, and the next model learns to agree with it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Noisy labels — set in haste, by a courtesy close, or with a typo | Validation plateaus below an unexplained ceiling; confident errors concentrate in particular classes | The recorded label disagrees with the true answer at some rate, systematically for some classes | Blind relabel a sample; report agreement as the ceiling; remove or down-weight systematic sources |
| Delayed labels — outcome arrives weeks later | Recent months show far fewer positives; model learns recency is safe | Unclosed label windows treated as negative | Exclude rows until the window closes; never map "unknown yet" to negative |
| Definition drift — reorg, policy change, new dispute rules | Step change in class prevalence at a date; old classes that no longer exist | The label means different things in different parts of the dataset | Version the label definition; rebuild from events; train on one version or add the version as a feature with care |
| Annotator disagreement — the task is genuinely ambiguous | Low inter-annotator agreement; model errors look reasonable to a human | No single true label exists for some examples | Keep multiple labels or soft labels; evaluate against agreement, not a forced majority |
| Model-generated labels — a previous router or model set them | New model reproduces the old system's known mistakes with a higher score | The training signal is the previous decision, lightly edited | Tag label provenance; evaluate on human-only labels; sample blind human review — see Feedback Loops |
The label as versioned, tested code
A label that lives as a mutable column set by whoever closed the ticket has no definition, no version and no test. The alternative is to derive it from raw events under a stated rule, so that the rule can be reviewed, versioned when the org changes, and tested on fixtures that encode the edge cases.
The SQL below defines the routing label as the team of the *first* assignment that was not subsequently reassigned, under a team mapping valid at the ticket's creation date. The two things to notice are the date-valid team mapping and the closed-window filter.
1-- label_v3: team of the first assignment that stuck (no later reassignment),2-- mapped through the org structure that was valid when the ticket was created.3WITH assignments AS (4 SELECT ticket_id, team_id, assigned_at,5 LEAD(assigned_at) OVER (PARTITION BY ticket_id ORDER BY assigned_at) AS next_assigned_at6 FROM ticket_assignment_events7),8stuck AS (9 SELECT ticket_id, team_id10 FROM assignments11 WHERE next_assigned_at IS NULL -- last assignment = the one that stuck12),13mapped AS (14 SELECT t.ticket_id,15 m.current_team_id AS label_team16 FROM tickets t17 JOIN stuck s USING (ticket_id)18 JOIN team_mapping m19 ON m.historical_team_id = s.team_id20 AND t.created_at >= m.valid_from AND t.created_at < m.valid_to21)22SELECT ticket_id, label_team, 'v3' AS label_version23FROM mapped24JOIN tickets USING (ticket_id)25WHERE tickets.closed_at IS NOT NULL26 AND tickets.closed_at < now() - interval '14 days'; -- reopen window has passedThe version string is not decoration. A metric reported against label_version = 'v2' is a different metric from one against v3, and a dataset that mixes them is measuring two definitions at once.
The ceiling, and what stays true after deployment
A blind relabelling of a few hundred tickets by experts, compared to the recorded labels, gives an agreement rate. That rate is roughly the best any model can score against the recorded labels, and a validation score at or above it is measuring the noise. The number belongs on the dashboard next to the metric it bounds.
After deployment, the label process keeps running — agents keep closing tickets, and now the model's route is the default they edit. The assumption that has to hold is that the labels still point at the truth and not at the model.
Labels arriving after deployment are produced by a process independent enough of the model that agreement with them still measures routing quality, and the label definition has not changed under the model.
holds when Label provenance is recorded; a blind human-review sample is drawn regularly with the model's output hidden; the label definition is versioned and a change triggers a rebuild and re-evaluation.
breaks when Agents accept the model's route by default and correct only egregious errors; a reorg remaps teams without a label version bump; closing incentives change and courtesy-closes rise.
respond Restore an independent label source — blind review, or a held-out slice routed by a rule — before trusting any metric, and bump the label version if the definition moved.
How to build it
Most important first.
- Write the label as versioned, tested code with a stated definition — "the team that resolved the ticket without further reassignment, under the current org structure" — and rebuild labels from raw events rather than from a mutable column (Label Construction, Data & Feature Tests).
- Measure label noise directly: have experts relabel a random sample blind and compute agreement with the recorded labels, overall and per team. That number is the model's ceiling and belongs beside every metric.
- Separate the sources of labels — human, previous model, rule, bulk remap — as a column, and evaluate on human-only labels even if training uses all of them.
- Handle delay explicitly: exclude rows whose label window has not closed, and never let "no label yet" become "negative" (Ground-Truth Delay).
- Where annotators legitimately disagree, keep the disagreement — multiple labels, or a soft label — rather than forcing a majority vote that hides the ambiguity.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Inter-annotator agreement on a blind relabelled sample, per class. This is the number that bounds achievable accuracy; a validation score above it is measuring agreement with noise.
- The share of training labels that originated from a rule or a previous model, and the model's agreement with those versus with human labels.
- Label prevalence per class by month, which shows definition drift and delay as step changes, and which no accuracy figure surfaces.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The label definition in production — what counts as a correct route, a fraud dispute, a churn — is the one the model was trained on, and a change to it triggers a label rebuild rather than a silent remap.
- The noise rate and its structure in incoming labels stay close to the measured rate on the relabelled sample, so the training signal continues to point at the truth.
- Labels produced by the deployed model itself are marked as such and are not fed back as human ground truth.
- Offline: relabel a stratified sample blind and compute agreement; run the label-construction code's tests against hand-built fixtures for each edge case — reassigned tickets, merged teams, unclosed windows.
- Online: sample production decisions for human review with the model's prediction hidden, and track agreement with the reviewer over time as a noise-free quality signal.
- Over time: monitor per-class label prevalence by month and the share of labels originating from the model; a step change in either is a label problem before it is a model problem.
What can go wrong
- The expert relabelling is done once; labels continue to be produced by agents under a new incentive — close tickets fast — and the noise rate drifts upward without a signal.
- Labels are rebuilt from events under the current org structure, and the remap for pre-merge tickets is a guess that is now versioned, tested, and confidently wrong.
- The model is deployed as the router, agents fix only its worst mistakes, and next year's labels are this model's outputs with light editing.
- Blind expert relabelling costs expert time on an ongoing basis, and a sample small enough to afford gives a noisy estimate of the ceiling.
- Rebuilding labels from raw events under a versioned definition means keeping the raw events and maintaining the label code through every org change.
- Evaluating only on human labels shrinks the evaluation set, sometimes below the size where the metric can distinguish candidates.
- "Validation accuracy is above what the agents achieve, so the model is better than the agents." The model is scored against the agents' labels. It cannot be more right than the labels; it can only agree with them more often than an agent would agree with another agent.
- "We cleaned the labels by remapping old teams to new ones." A remap is a new label definition applied retroactively to tickets that were never handled under it. It is versioned drift, not cleaning.
- "More labels will fix the noise." More labels from the same process have the same noise. The ceiling moves only when the process that produces the labels changes.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model trained against noisy labels has its offline metric bounded by the label noise is true for every supervised task and model family.
- TASK-SPECIFICThe delay and definition-drift failures dominate for outcome labels observed after a window — churn, default, dispute; annotator disagreement dominates for judgement labels — sentiment, relevance, routing; model-generated labels dominate wherever a previous system already made the decision.
Where the depth lives
This domain teaches the model and hands the rest off by name.