Data Poisoning
A corrupt training source teaches the model wrong behaviour, and the offline metric — computed on the same corrupt data — approves. Provenance, validation, trusted pipelines and slice evaluation are the defence.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A model retrained on data that users can influence started behaving strangely for one category. How does a corrupt source reach the weights, and what would have shown it?
Our marketplace uses a model to decide which listings get flagged for review. Sellers can dispute a flag, and disputes that succeed become "not fraudulent" labels for the next retrain. A cluster of sellers in one category has been disputing everything, winning often because reviewers are overloaded, and the retrained model has stopped flagging that category. Nobody attacked our servers. Our own feedback loop did this.
More labels are better. Disputes are free labels from the people who know the listing best; merge them in and retrain. The validation metric went up after the merge, which confirms it.
The disputed labels say "not fraud" for listings that were flagged as fraud, in one category, from a correlated group of sellers. The model learns that the category's fraud-like features are not fraud. It is learning exactly what the data says.
- The disputed labels say "not fraud" for listings that were flagged as fraud, in one category, from a correlated group of sellers. The model learns that the category's fraud-like features are not fraud. It is learning exactly what the data says.
- The validation metric improved because the held-out set contains the same poisoned labels; the model is rewarded for agreeing with the corruption. Offline evaluation cannot detect poisoning that is present on both sides of a random split (Evaluation Leakage is the same shape from a different cause).
- Because the shift is confined to a category, the aggregate metric barely moves and the slice that collapsed was never evaluated on its own (Evaluation Slices). The signal was chargebacks in that category, which arrive months later (Ground-Truth Delay).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a listing is fraudulent; the label comes from reviewer outcomes and from resolved disputes. The defensive target is that a training source which users can influence cannot shift the model's behaviour without a validation gate or a slice evaluation noticing.
- Listings with seller features, text and images; labels from three sources: reviewer decisions, resolved disputes, and chargebacks. Disputes are the source users control most directly, and the one with the weakest reviewer attention.
- The retrain merges all three label sources into one column. Provenance is not recorded per row. Evaluation is on a random held-out split of the merged data, so the poisoned labels appear on both sides of the split.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The chain is: Corrupt Training Source → Model Learns Wrong Behaviour. A model minimises loss on the labels it is given. If a subset of labels is systematically wrong in a way correlated with features, the model learns that correlation as if it were real. There is no distinction, from inside training, between a true pattern and a consistently mislabelled one (Label Quality).
- The source of corruption can be accidental — a broken join, a labelling vendor's drift — or influenced: a public dataset that anyone can contribute to, scraped content, or a feedback loop where users affect their own labels (Feedback Loops). The defensive question is the same for all of them: where did each row come from, and can whoever controls that source shift the model?
- Detection is by comparison against something the source cannot touch. A trusted held-out set labelled by a different process; a slice evaluation that isolates the influenced source or the affected category; a validation gate that flags a label-distribution shift per source between retrains.
Corrupt source, wrong behaviour, approving metric
The pipeline device traces how a source reaches the weights and where the corruption is visible along the way. It is visible early — at ingestion, per source, as a distribution shift — and invisible late, once the sources are merged and the split is random. Every step that merges or randomises destroys the information that would have caught it.
The defence is therefore mostly about preserving information: keep the source on the row, keep an evaluation set the source cannot touch, keep the slices the business cares about separate in the metric.
- 1Source
Disputes resolved by overloaded reviewers become "not fraud" labels
fails by A correlated group of sellers in one category disputes everything and wins often
- 2Ingestion
Labels from reviewers, disputes and chargebacks collected
fails by No per-source label-distribution check; the dispute source's shift in one category passes
- 3Merge
Three sources become one label column
fails by Provenance is dropped; the poisoned rows are now indistinguishable
- 4Split
Random held-out split of the merged table
fails by Poisoned rows land on both sides; validation now rewards agreeing with them
- 5Train
Minimise loss on the labels as given
fails by Learns that the category's fraud-like features are "not fraud" — exactly as the data says
- 6Evaluate
Aggregate metric on the held-out split
fails by Slightly improved; the collapsed category is a small share of the aggregate
- 7Deploy
Model stops flagging the category
fails by The first honest signal is chargebacks, months later
The leakage simulator at /ml/leakage shows the same structural fact from the other direction: a validation set that shares the corruption with training approves it. Poisoning is a label problem rather than a feature problem, and the split is where it hides.
Why the validation set could not see it
A leakage device is the right tool even though nothing here is leakage in the temporal sense. The "feature" is the label source itself: a dispute outcome that looks like a high-quality label and is in fact a signal the sellers control. It behaves like leakage in the way that matters — it makes the offline metric confident about something false — and it has the same cure: evaluate against information the source cannot reach.
The fine case matters. Dispute outcomes are legitimate labels when the review process is independent and not overloaded; the problem is not the source but the source's controllability, which changes over time.
looks like A "not fraud" label from a resolved dispute, stored in the same column as reviewer decisions and chargebacks, with no marker of where it came from.
why it leaks The sellers who produce the listings also produce the disputes; when review is overloaded, the label reflects their persistence rather than the listing's nature. The model learns the correlation between their features and "not fraud" as if it were real.
fix Keep provenance per row; gate per-source label distributions between retrains; evaluate on a trusted set labelled independently, per slice; weight or quarantine influenced sources for training.
The gate that would have caught it
A per-source, per-category label-distribution check between retrains is cheap and would have fired here on the first poisoned retrain. It compares each source's label rate in each category against the previous training set and flags a shift beyond a band. It is a data test, not a model test, and it belongs in the validation stage of the pipeline.
The assumption device names what has to remain true for the gate to keep working: that the source is still on the row when the gate runs, and that someone looks when it fires.
A shift in the labels produced by a user-influenced source is caught by the validation gate or the trusted-slice evaluation before a candidate deploys.
holds when Provenance reaches the training table; the per-source gate runs on every retrain; the trusted set is independent, refreshed and large enough per critical slice.
breaks when A pipeline refactor merges sources before the gate; the trusted set is topped up with dispute outcomes "to get more data"; the shift is slow enough to stay inside the band across many retrains.
respond Quarantine the source for the affected categories, retrain from the last clean snapshot, and fix the review process that made the source controllable — the model is the symptom.
1WITH rates AS (2 SELECT snapshot, label_source, category,3 AVG(CASE WHEN label = 'not_fraud' THEN 1.0 ELSE 0.0 END) AS not_fraud_rate,4 COUNT(*) AS n5 FROM training_labels6 WHERE snapshot IN ('v41', 'v42') -- previous and candidate training sets7 GROUP BY snapshot, label_source, category8)9SELECT a.label_source, a.category,10 a.not_fraud_rate AS prev_rate, b.not_fraud_rate AS new_rate, b.n11FROM rates a JOIN rates b12 ON a.label_source = b.label_source AND a.category = b.category13WHERE a.snapshot = 'v41' AND b.snapshot = 'v42'14 AND b.n >= 20015 AND ABS(b.not_fraud_rate - a.not_fraud_rate) > 0.15; -- band is a policy, not a constantThe query only works because label_source survived to the training table. The merge step that drops it is the one to fight for.
How to build it
Most important first.
- Record provenance per row: which source produced the label, when, through what process. Provenance is what lets you evaluate per source and roll a source back (Dataset Versioning, Model Lineage).
- Validate label distributions per source and per category between retrains: a source whose "not fraud" rate jumped in one category is a gate failure, not a training input (Data & Feature Tests).
- Hold out a trusted evaluation set labelled by a process users cannot influence — reviewer decisions with fresh eyes, or chargebacks — and evaluate every candidate on it, per slice. The influenced source may be in training; it must not be the judge.
- Down-weight or quarantine influenced sources: a dispute becomes a label only after a reviewer confirms it, or is weighted below a reviewer decision, or is excluded for categories where dispute rates are anomalous.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-source, per-category label distribution over time — the number that shows a source shifting before the model does.
- Candidate quality on the trusted held-out set, per slice, compared to the incumbent. A candidate that improves on the merged validation set and regresses on the trusted slice is poisoned or overfitted, and either way does not ship.
- The merged validation metric is the number that looks relevant and is not: it is computed on the source under suspicion.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every label row carries its source and that source's trust level survives to the training set, so evaluation and rollback per source are possible.
- The trusted evaluation set is produced by a process that the influenced sources cannot affect, is refreshed, and is large enough per critical slice to detect a regression.
- Label distributions per source and category are compared between retrains, and a shift blocks training until someone has looked.
- Offline: inject a synthetic label shift into one source for one category in a staging retrain and confirm the validation gate blocks it and the trusted-slice evaluation shows the regression.
- Online: after each retrain, compare flag rates per category against the previous model on live traffic; a category whose flag rate collapsed is a slice to investigate before the chargebacks arrive.
- Over time: when delayed ground truth arrives, compute quality per source of training label. A source whose labels disagree with ground truth more than the others is the one to down-weight.
What can go wrong
- The trusted set is small, so the per-slice comparison is noisy and a real regression is inside the confidence interval (Metric Uncertainty).
- The provenance is recorded at ingestion and lost at the feature pipeline, which merges sources into one table before training reads it.
- The quarantine rule is keyed on dispute rate per category, and the influence moves to a different category or a slower rate below the threshold.
- Provenance per row is a schema change through the whole data path, and the feature pipeline is the place it will be dropped.
- A trusted held-out set labelled by an independent process is expensive and small, which is why it must be used for judgement and not for training.
- Quarantining influenced sources throws away real signal; disputes that are honest are exactly the labels that correct reviewer mistakes.
- "Validation improved after adding disputes, so they helped." Validation was computed on data that includes the disputes. It measures agreement with the source, not correctness.
- "Nobody attacked us, so this is not a security issue." The mechanism does not care about intent. A feedback loop that lets users influence their own labels is the same exposure as a hostile contributor to a public dataset.
- "Retrain without the bad category." That removes the symptom for one category and leaves the feedback loop that produced it; the next category will be next quarter.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model learns systematically wrong labels as if they were right, and that a random split of poisoned data validates the poison, follows from what training does and holds for every model family.
- DOMAIN-SPECIFICMarketplaces, spam, content moderation and fraud have users who both generate inputs and influence labels, so the feedback-loop variant dominates; a demand-forecasting model on internal sales data faces poisoning mainly through broken joins and vendor labelling drift, which the same provenance and slice controls catch.
- CONTESTEDA serious position holds that user-influenced labels are too valuable to quarantine — disputes catch reviewer errors at scale, and excluding them makes the model worse on average than a small poisoned slice makes it. That is often right on aggregate quality; the reply is that the influenced source can still be used for training if it is never used for evaluation and its distribution is gated, which keeps the signal and removes the blind spot.
Where the depth lives
This domain teaches the model and hands the rest off by name.