Dataset Versioning
A table name is not a version. A dataset the model trained on must be an immutable snapshot with an identifier that resolves to the same rows forever — or the run record points at nothing.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How is a training dataset versioned so that a run record resolves to exactly the rows the model saw, and what breaks when the version is a name?
A model trained "on the March data" needs to be retrained with one bug fixed in the code. The engineer points the job at the same table. The metric moves in a direction the bug fix cannot explain. Somebody backfilled the table two weeks ago and nobody can say what the March data was any more.
Record the table name and the date range. That is what the query used, so that is the dataset. Re-running the query gives the same data.
The table was backfilled: a source correction rewrote two partitions inside the date range. The same query returns different rows and the run cannot be reproduced (Reprocessing vs Retrying).
- The table was backfilled: a source correction rewrote two partitions inside the date range. The same query returns different rows and the run cannot be reproduced (Reprocessing vs Retrying).
- The label job was fixed and re-run over history. Every label in the range changed, the schema did not, and nothing in the run record shows it — the metric moves and the model is blamed.
- Late-arriving events were appended to old partitions. The "March data" grew by a few percent after the model was trained, and the newer rows are systematically different — they are the events that arrive late (Late-Arriving Data).
- A column was renamed and a compatibility view was added, so the query still runs and returns a column with the same name and a subtly different definition (Schema Evolution).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts whatever it predicts; the versioning target is that a dataset identifier in a run record resolves to exactly the same examples, labels and schema at any later time.
- A version is a claim about immutability, not a name. "Dataset v12" means "these rows, this schema, this label definition, and nothing else will ever answer to v12".
- Training data assembled from warehouse tables that are themselves maintained by a data platform: partitions appended daily, backfilled when a source is corrected, and occasionally rewritten when a transformation changes (Backfills).
- Labels produced by a separate job with its own fixes: a label definition that changed in March, applied retroactively to all history.
- A feature pipeline that reads the dataset and produces the training matrix; its own version is a separate concern (Feature and Model Versioning).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A version is an immutable snapshot plus an identifier that can only ever refer to that snapshot. The strongest identifier is a content hash: a digest of the rows (or of a manifest of the files that hold them), so that any change to any row produces a different identifier. The second strongest is a partition manifest — an explicit list of file paths with their digests — which is what table formats with snapshot isolation provide (Open Table Formats).
- Immutability has to be enforced by the storage, not promised by convention. A snapshot in an object store with overwrite disabled, or a table format's committed snapshot id, is immutable; a "do not touch this table" note is not.
- Labels are part of the dataset version, not a property of the model: Dataset v12 with Labels v3 is a different dataset from v12 with Labels v2, and a run record has to name both. Lineage from raw data to the snapshot is what lets the snapshot be rebuilt or audited (Data Lineage).
Four versions, one model
A production model is the product of at least four versioned inputs: the dataset, the labels applied to it, the feature pipeline that transformed it, and the model artifact itself. Each has its own lifecycle and its own owner; a run record has to name all four, and each name has to be immutable.
The dataset and label versions are the ones the data platform owns and the ML team depends on. This lesson is about those two; the feature and model versions follow (Feature and Model Versioning).
A manifest that changes when the data does
The identifier has to be a function of the content. A manifest that lists every file the snapshot consists of, with each file's digest, hashed together, changes whenever any file changes — including an in-place rewrite that keeps the path. A list of paths alone does not.
Labels are included in the manifest as their own files with their own digests, so a label fix that rewrites the label file changes the dataset identifier even if the feature rows are untouched.
1import hashlib, json2 3def file_digest(path, chunk=1 << 20):4 h = hashlib.sha256()5 with open(path, "rb") as f:6 for block in iter(lambda: f.read(chunk), b""):7 h.update(block)8 return h.hexdigest()9 10def dataset_version(row_files, label_files, label_definition_version, schema):11 manifest = {12 "rows": {p: file_digest(p) for p in sorted(row_files)},13 "labels": {p: file_digest(p) for p in sorted(label_files)},14 "label_definition": label_definition_version,15 "schema": schema, # a renamed column is a new version16 }17 blob = json.dumps(manifest, sort_keys=True).encode()18 return hashlib.sha256(blob).hexdigest(), manifest19# Store the manifest next to the files, immutable. The run record carries20# the hash; a later reader recomputes it and knows whether anything moved.The digests are over contents, not paths, so a backfill that rewrites a partition in place changes the version. The schema is in the manifest so a compatibility view that keeps a column name over a new definition also changes it — provided the schema captured is the definition, not just the name.
What must stay true for the version to be a version
The identifier is only as good as the storage behind it. A content hash proves whether the data changed; it does not prevent it. Prevention is the storage's job — overwrite disabled, retention tied to the registry — and it is the part that has to be negotiated with whoever owns the platform.
The failure to plan for is silent: a snapshot that was mutated and is still referenced by a run record that says nothing is wrong. The detector is recomputing the hash, on a schedule, before anyone needs the answer.
The rows, labels and schema behind a dataset version cannot change, and the snapshot is retained for at least as long as any model trained on it is in production or subject to audit.
holds when Snapshots live in storage with overwrite disabled; retention is driven by registry references rather than by age; the content hash is recomputed on a schedule and compared.
breaks when A cleanup job refreshes or deletes snapshots; the snapshot is a view over a live table; labels are joined at training time from a mutable source; retention expires a snapshot the production model was trained on.
respond Restore from raw history using the recorded query and lineage if the platform kept it; otherwise mark every model trained on the snapshot as non-reproducible and prioritise retraining on a properly snapshotted dataset.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Backfill rewrites two partitions | Rerun metric moves with no code change | The run record named a table and date range | Snapshot with a content hash; record the hash |
| Label fix applied to history | Every model looks worse or better at once | Labels joined live; no label version recorded | Version labels; a fix is a new version, never a rewrite |
| Late events appended to old partitions | The "same" dataset is larger next month | Date partition treated as immutable | Snapshot at a stated ingestion cutoff; record the cutoff |
| Retention deletes a 90-day-old snapshot | Production model cannot be reproduced for an audit | Retention by age, not by registry reference | Tie retention to live registry entries |
How to build it
Most important first.
- Materialise the training set as a snapshot: write the exact rows used to an immutable location, compute a content hash or a file manifest with digests, and record that identifier — never the query — in the run (Experiment Tracking).
- Version labels separately and explicitly: a label definition version and the date it was applied, recorded alongside the dataset version.
- Use the data platform's snapshot mechanism where it exists — a table format's snapshot id, a partition manifest, a copy-on-write export — and check that it is immutable rather than assuming.
- Keep the query and its lineage as documentation of how the snapshot was produced, so it can be rebuilt from raw history if the platform enforces raw retention (Keeping Raw History: The Recovery Position and the Liability).
- Treat schema as part of the version: a renamed or redefined column is a new dataset version, even when the query still runs.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Whether the dataset identifier in any run record resolves to a snapshot whose content hash matches the recorded one. This is binary and is the whole point.
- The fraction of production models whose training dataset version can be resolved and re-read today.
- Do not measure "the table exists". A table that exists and has changed is worse than a missing one, because it will be trusted.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The storage holding a snapshot enforces immutability — no overwrite, no in-place rewrite, no retention deletion — for as long as any model trained on it might need to be reproduced or audited.
- The identifier is a function of the content: any change to rows, labels or schema yields a new identifier, so a mismatch is detectable by recomputing the hash.
- The label version is recorded with the dataset version, and a label fix produces a new version rather than a rewrite.
- Offline: recompute the content hash of a sample of snapshots referenced by run records and compare; a mismatch is a mutation that has already happened.
- Online: at promotion, resolve the candidate's dataset and label versions and fail if either does not resolve or does not hash-match (Promotion Is a Checklist, Not a Score).
- Over time: a retention check that no snapshot referenced by a live or recent registry entry is eligible for deletion.
What can go wrong
- The snapshot is a copy in object storage with overwrite enabled, and a cleanup job "refreshes" it.
- The content hash is computed over the file list, not the contents, and an in-place rewrite of one file leaves the hash unchanged.
- Snapshots are versioned but labels are joined at training time from a live table, so the dataset is immutable and the labels are not.
- Retention deletes old snapshots after ninety days, and the production model is a hundred days old.
- Materialised snapshots duplicate storage — every training run that touches a fresh snapshot keeps a copy — and retention becomes a negotiation between cost and auditability.
- Content hashing large datasets is a real compute cost at snapshot time; a manifest of file digests is cheaper and slightly weaker.
- Immutability moves label fixes from "rewrite history" to "new version", which is the right thing and also means old versions with the wrong labels remain and must be understood as such.
- "We version the query, so we version the data." The query is a function; the data is its output at one moment. The same query on the same table next month returns the rows the platform has since corrected, appended or rewritten.
- "The table is partitioned by date, so March is fixed." Partitions are rewritten by backfills and appended to by late events. A date partition is a location, not a version.
- "Labels are the model's concern, not the dataset's." A model trained on Labels v2 and one trained on v3 learned different things from the same rows. The label version is a dataset property and the run record must carry it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvery model trained on data that a platform maintains has this problem, whatever the task; the details differ — a streaming source needs an offset-based snapshot, a warehouse a table snapshot — but "a name is not a version" is universal.
- FRAMEWORK-SPECIFICOpen table formats with snapshot isolation give a snapshot id that is immutable by construction and cheap to reference; on a plain file store the ML team has to build the manifest and enforce the immutability themselves, and the enforcement is the part usually skipped.
- CONTESTEDA serious position holds that snapshotting every training set is wasteful and that recording the query plus the table format's snapshot id at query time is enough, since the platform already keeps history. The counter is that platform retention is set for the platform's needs, not the model's, and that a snapshot id the platform has expired resolves to nothing — the ML team needs retention tied to the registry, which usually means its own copy.
Where the depth lives
This domain teaches the model and hands the rest off by name.