ML Privacy
A training set is personal data, an artifact can memorise it, and a prediction log is a record of people. Privacy is a design property of the pipeline — minimisation, retention, access, and honest limits on anonymisation.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model needs the data to learn and the logs to be monitored. What personal data does the system hold, where, for how long, and who can get it back out?
A health-adjacent app trains a model on user activity logs to predict who will stop using the app. A deletion request has arrived from a user under a data-protection law, and the team has realised that the user's rows are in the raw logs, three feature tables, two training snapshots, the prediction log, and possibly the weights.
Hash the user id before training, drop the name and email, and call the dataset anonymised. The model only sees numbers. Keep everything, because storage is cheap and someone might need to reproduce a run.
A hashed id with a weekly activity pattern and a coarse location is re-identifiable by joining to any other dataset that has the same pattern. Removing direct identifiers is pseudonymisation, and the rest of the row is a fingerprint.
- A hashed id with a weekly activity pattern and a coarse location is re-identifiable by joining to any other dataset that has the same pattern. Removing direct identifiers is pseudonymisation, and the rest of the row is a fingerprint.
- A model with enough capacity memorises rare training rows. Membership inference — asking whether a specific record was in the training set — works on such models, and extraction of memorised sequences has been demonstrated on large generative models (ML Security).
- The deletion request cannot be honoured because nobody knows where the copies are, and the training snapshot that reproduces the current production model contains the row. Deleting it breaks reproducibility; keeping it breaks the law.
- The prediction log, kept for monitoring, is the most complete per-person record in the company and is readable by everyone with dashboard access.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict thirty-day disengagement from activity history. The label is the absence of activity, which is itself a fact about a person.
- The privacy target of the surrounding system is that personal data is held only where necessary, only as long as necessary, accessible only to those who need it, and removable when the person asks.
- One example is one user-week of activity features — session counts, feature usage, time of day — joined to profile attributes. Every row is about an identifiable person.
- Training snapshots are copied to object storage for reproducibility. Feature tables are rebuilt nightly. The prediction log stores the feature vector and the score for every request.
- Copies multiply: notebooks, an evaluation dataset on a laptop, a vendor's labelling tool.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Data minimisation: collect and retain only the fields the model has been shown to need. Every feature is a liability with a benefit, and features with no measured contribution are pure liability.
- Retention: each store — raw, features, snapshots, logs, artifacts — has a stated retention period, and the training pipeline is designed so that deleting a row from the source propagates, which means snapshots are rebuilt or expire rather than being kept forever.
- Anonymisation is a spectrum with limits. Removing identifiers is pseudonymisation; k-anonymity and aggregation reduce re-identification but degrade under joins; differential privacy bounds what any output can reveal about any single record at a quantified cost to accuracy, and is the only one with a formal guarantee.
- Access control: the training set, the feature store and the prediction log are personal data stores and need the same role-based access and audit logging as the user database.
Where the person is
A deletion request is a search. The rows are in the raw logs, in the nightly feature tables, in every training snapshot copied to object storage, in the prediction log, in the evaluation set someone exported, and — for rare rows and a large model — in the weights. The pipeline diagram most teams draw shows data flowing forward; the privacy diagram shows where it stays.
Each store needs an owner, a retention period, an access policy and a deletion path. The inventory is not documentation; it is the mechanism by which a request can be honoured at all.
- 1Raw event log
Every activity event with user id and timestamp; retained for reprocessing.
fails by Retained forever because "we might need to backfill"; the deletion path is a manual job nobody runs.
- 2Feature tables
Nightly aggregates per user; rebuilt from raw.
fails by Rebuilt from raw, so deletion propagates — unless a feature is cached in a serving store that is not rebuilt.
- 3Training snapshots
A frozen copy per training run, for reproducibility.
fails by Copies accumulate; the snapshot behind the production model is kept indefinitely and contains every deleted user.
- 4Model artifact
The weights, and any preprocessing state such as vocabularies and category maps.
fails by Memorised rare rows; a category map that literally contains user-entered strings.
- 5Prediction log
Feature vector, score, version, per request, for monitoring and debugging.
fails by The most complete per-person record in the company, readable by anyone with dashboard access.
The deletion test is to submit a synthetic request and check each of these afterwards. Most teams discover the third and fifth rows on the first attempt.
Anonymisation and its limits
Removing the name and hashing the id produces a table that still contains a person's week — when they open the app, for how long, from roughly where. Anyone holding another dataset with that pattern can join. This is not a theoretical attack; it is how most published re-identifications were done.
The techniques form a ladder. Pseudonymisation removes direct identifiers. Generalisation and k-anonymity coarsen quasi-identifiers until each combination is shared by k people, and degrade under a join with any dataset that has finer values. Differential privacy adds calibrated noise to the training process so that the presence or absence of any single record changes the output distribution by at most a bounded factor, and the bound is the guarantee. Only the last one survives an adversary with auxiliary data, and it costs accuracy.
| Technique | What it removes | What it does not protect against | Cost |
|---|---|---|---|
| Pseudonymisation (hash the id) | Direct identifiers | Re-identification by joining on the remaining attributes | None — which is why it is not protection |
| Generalisation / k-anonymity | Uniqueness of quasi-identifier combinations | Joins with finer auxiliary data; homogeneity within a group | Feature resolution |
| Aggregation only | Individual rows | Differencing attacks between overlapping aggregates | The model cannot be trained on individuals |
| Differential privacy in training | Bounded influence of any one record on the output | Nothing, within the budget — the guarantee is formal | Accuracy, especially on rare rows; a budget to manage |
What must stay true after deployment
The privacy design is a set of claims — this data is held here, for this long, by these people — and every one of them decays. A new notebook copies the training set. A feature is added. A dashboard is shared. The assumption to monitor is that the inventory still describes the system.
The prediction log deserves its own note: it is needed for monitoring, it holds personal data, and both facts are true. The resolution is to log what monitoring needs, protect it like the user database, and expire it.
Every store that holds personal data from the pipeline is listed, has a retention period that is enforced, and is reachable by the deletion path.
holds when Snapshots and logs are created only by the pipeline through a registry that records them; ad-hoc exports are blocked or expire; the deletion test passes on schedule.
breaks when An engineer exports the evaluation set to a laptop; a vendor labelling tool receives raw rows; a new serving cache stores feature vectors that the rebuild does not touch.
respond Add the store to the inventory with an owner and a retention, or remove it; treat a failed deletion test as a production incident.
1-- After processing a deletion request for :user_id, every one of these must be zero.2SELECT 'raw_events' AS store, count(*) FROM raw_events WHERE user_id = :user_id3UNION ALL4SELECT 'features_daily', count(*) FROM features_daily WHERE user_id = :user_id5UNION ALL6SELECT 'training_snapshots', count(*) FROM training_snapshots WHERE user_id = :user_id7UNION ALL8SELECT 'prediction_log', count(*) FROM prediction_log WHERE user_id_hash = :user_id_hash9UNION ALL10SELECT 'serving_cache', count(*) FROM serving_feature_cache WHERE user_id = :user_id;The training-snapshot row is the one that fails. The honest choices are to rebuild the snapshot without the row and accept that the old model is no longer exactly reproducible, or to let the snapshot expire on its retention date and record that the model will be retrained by then.
How to build it
Most important first.
- Inventory: list every place personal data lands in the pipeline, including derived artifacts. A deletion request is the test of the inventory.
- Minimise at the feature level with evidence — a feature that permutation importance shows to be inert is removed, and profile attributes not needed for the prediction never enter the feature table.
- Set retention per store with expiry enforced by the platform, and design reproducibility around versioned dataset definitions plus expiring snapshots rather than permanent copies (Dataset Versioning).
- Log for monitoring what monitoring needs: the feature vector without the identifier where possible, hashed and salted ids where a join back is required, and the score. Restrict and audit access to the log (Prediction Logging).
- Where the data is sensitive enough, train with differential privacy and accept the measured accuracy cost; where it is not, say so in the design and why.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The number of stores holding a given person's data, and the time to complete a deletion across all of them. This is the number a regulator will ask for.
- Per-feature contribution, so minimisation is an argument rather than a guess.
- For models trained with differential privacy, the privacy budget spent and the accuracy delta against the non-private model, because the trade is the design decision.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The data inventory is complete — no new copy of the training set has been made in a notebook, an evaluation job or a vendor tool since it was last checked.
- Retention expiry actually runs, on every store, and a snapshot older than its period is gone rather than merely marked.
- The features in the model are the ones the minimisation review approved; a feature added for accuracy has been through the same review.
- Offline: run a synthetic deletion request through the pipeline and confirm the row is absent from every store in the inventory afterwards, including the next training snapshot.
- Online: audit log queries showing who read the prediction log and the feature store, reviewed on the same cadence as production database access.
- Over time: a membership-inference test against each promoted model on a held-out set of known training rows, to measure how much the artifact leaks.
What can go wrong
- The deletion propagates to the source tables and the nightly feature rebuild, but not to the snapshot that the production model was trained from, so the person is still in the weights and the audit trail.
- The prediction log is anonymised by dropping the id, and monitoring can no longer join outcomes back to predictions, so quality can no longer be measured (Ground-Truth Delay).
- A differentially private model is trained and the epsilon is set so high to recover accuracy that the guarantee is meaningless while the label on the design document still says "private".
- Expiring snapshots means an old model cannot be exactly retrained, which is a real loss for incident investigation.
- Differential privacy costs accuracy, and the cost is largest on the rare rows — which are often the ones the model most needs to get right.
- Minimisation removes features that might have helped, and the argument for removing them is made on a dataset that is itself changing.
- "We hashed the ids, so it is anonymous." A stable hash is a pseudonym, and the rest of the row identifies the person. Anonymisation is a property of what can be inferred, not of which column was removed.
- "The model only has weights, it does not contain the data." Weights can encode training rows, and for rare rows and high-capacity models they demonstrably do.
- "Logs are operational data, not personal data." A log that records a feature vector and a score for an identifiable request is a record about a person, whatever it was kept for.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICHealth, finance and children's data carry legal obligations that make differential privacy and strict minimisation mandatory; a recommendation model on public catalogue interactions carries far less, and the same controls would be disproportionate.
- CONTESTEDA serious position holds that differential privacy is the only anonymisation with a guarantee and should be the default for any model trained on personal data, since every weaker technique has been broken by a join. The counter-position is that at the privacy budgets that give a meaningful guarantee the accuracy cost is severe, especially on minority subgroups, so the practical protection comes from minimisation, retention and access control — and that a private model trained on data the team should not have collected is not a privacy win. Both agree that "we hashed the id" is neither.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Data-protection law — what counts as personal data, what a lawful basis is and how long a deletion may take are legal questions; this lesson gives the pipeline design that makes the answers achievable.