Reproducibility
Same code, same data, same seed, different number. Reproducibility is a property of the entire environment — kernels, data order, library versions, reduction order across workers — and a seed pins only one of them.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why does a rerun with the same seed produce a different metric, and what has to be pinned for a training run to be reproducible?
An auditor asks for the model behind a credit decision to be retrained from its recorded inputs and shown to produce the same scores. The team reruns the pipeline with the recorded seed. The validation metric is close but not equal and a few hundred applicants flip across the threshold. "Close" is not what the auditor asked for.
Set the seed everywhere — Python, NumPy, the framework — record it with the run, and rerun with the same seed. Randomness is the only reason two runs differ.
GPU kernels for some operations are non-deterministic by default: atomic adds accumulate in whatever order the hardware schedules, so the same inputs give a different low-order bit, and the difference compounds over ten thousand steps.
- GPU kernels for some operations are non-deterministic by default: atomic adds accumulate in whatever order the hardware schedules, so the same inputs give a different low-order bit, and the difference compounds over ten thousand steps.
- Data order changed: the dataset is sharded across four workers instead of two, so each worker's shuffle — seeded identically — walks a different subset, and the gradient at step one is already different.
- The framework's minor version changed a default: a different initialisation scheme, a different epsilon in the optimiser, a different reduction algorithm. The seed is the same; the function that consumes it is not.
- Floating-point addition is not associative. Summing gradients across workers in a different order — two workers versus four — gives a different sum in the last bits, and the model diverges from there (Gradient Synchronisation).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts default; the reproducibility target is that a rerun from the recorded inputs yields the same artifact, or an artifact whose predictions differ by an amount that is stated, bounded and explained.
- Bit-for-bit reproduction and statistical reproduction are different targets with different costs; the lesson is knowing which one you have.
- The run record: commit, dataset version, feature version, config, seed. All present, all correct.
- The environment: a Python version, a deep learning framework, a CUDA and cuDNN version, a GPU model, a number of data-loader workers, a number of training workers. Some of these were recorded and some were not.
- The rerun: on a different GPU, with a newer minor version of the framework, on a cluster where the dataset is sharded across four workers instead of two.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A training run is deterministic given every input, where "every input" includes the order in which floating-point operations are performed. Anything that changes that order — a different kernel, a different worker count, a different thread schedule, a different hardware reduction tree — changes the result in the low-order bits, and gradient descent amplifies low-order differences over steps.
- A seed fixes the pseudo-random sequences the code draws from: which examples go in which split, the initial weights, the shuffle order, the dropout masks. It does not fix which kernel the framework selects, how many threads reduce a sum, or which version of a library interprets the seed (Random Seeds).
- So reproducibility has layers: source, data, feature logic, configuration, dependencies, runtime, hardware, seeds. Bit-for-bit reproduction requires all of them and deterministic kernels, and costs speed. Statistical reproduction — the same metric within the seed-variance band — requires the first several and tolerates the rest.
The layers a seed does not reach
Each layer is an input, and a rerun that changes any of them is a different run. The seed sits at the bottom and is the one everybody sets; the layers above it are the ones that actually change between the original and the audit rerun.
The ML Failure Simulator at /ml/failures includes readings that are consistent with more than one cause; a rerun that fails to reproduce has the same property — a changed metric is consistent with any layer having moved — which is why the layers have to be pinned rather than diagnosed after the fact.
- 1Source
A commit, with a clean tree.
fails by A branch name; an uncommitted local change.
- 2Dataset
An immutable snapshot identified by content hash.
fails by A table name that a backfill rewrote (Dataset Versioning).
- 3Feature logic
A pinned feature-definition version.
fails by The same feature name with a changed window or null policy.
- 4Configuration
The resolved config after every override.
fails by The defaults file, with the overrides forgotten.
- 5Dependencies
A lock file; the framework's exact version.
fails by A minor upgrade that changed a default initialiser or optimiser epsilon.
- 6Runtime
A container image by digest; CUDA and cuDNN versions.
fails by A floating base-image tag rebuilt with a new CUDA.
- 7Hardware
GPU model, worker count, thread count, data sharding.
fails by Two workers instead of four: a different reduction order and a different shuffle walk.
- 8Seeds
Split, init, shuffle, dropout — each named.
fails by One global seed that a library reseeds internally.
Why floating point makes the worker count an input
Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. Summing a gradient across two workers and across four performs the additions in different orders and gives different sums. The difference is tiny at step one and is fed back into the weights, where ten thousand steps of gradient descent amplify it into a different model.
The same applies inside one GPU: kernels that accumulate with atomic operations add in whatever order the hardware schedules, which differs between runs on the same machine. Deterministic kernel modes trade this for speed by fixing the order.
1import numpy as np2 3rng = np.random.default_rng(0)4grads = rng.standard_normal(1_000_000).astype(np.float32)5 6two_workers = np.float32(grads[:500_000].sum()) + np.float32(grads[500_000:].sum())7four_workers = sum(np.float32(grads[i::4].sum()) for i in range(4))8sequential = np.float32(0)9for g in grads[:1000]: # even the first thousand, summed one by one10 sequential += g11print(two_workers, four_workers) # differ in the low-order bits12# Feed either into an optimiser step and the models diverge from here.Nothing in this snippet is random after the seed. The two sums differ because the additions were grouped differently, which is what changing the worker count does to gradient synchronisation. A seed cannot pin it; only fixing the grouping can.
Which reproducibility you have, and what must stay true
Two honest claims exist. "Bit-for-bit": the rerun produces the same artifact digest, which required deterministic kernels and fixed hardware and cost speed. "Statistical": the rerun's metric lands within the across-seed band, which required the inputs pinned and several seeds run. A team that has not chosen is claiming neither.
The assumption to make explicit is that a rerun executes the same computation. It is broken by any layer moving, and the detector is a reproduction test that runs before an auditor asks.
Given the recorded identifiers, a rerun performs the same floating-point operations in the same order (bit-for-bit) or the same computation up to a measured seed-variance band (statistical).
holds when Every layer is pinned by an immutable identifier; deterministic modes were enabled at the original run; worker, thread and sharding counts are recorded and reused; the band was estimated from enough seeds.
breaks when Hardware changes; a floating image tag is rebuilt; a library upgrade changes a default; the rerun uses a different worker count; determinism was enabled only for the rerun.
respond Find the layer that moved by bisecting inputs, not by rerunning with more seeds; record the reproducibility level and its band on the registry entry so the next reader does not have to rediscover it.
Seed recorded; everything else is whatever is current. The rerun differs and the difference is attributed to randomness.
Commit, dataset hash, feature version, resolved config, lock file, image digest, hardware class, worker counts and named seeds recorded; determinism mode recorded; the artifact's registry entry states "bit-for-bit" or "statistical, band ±x".
The seed only fixes the random draws. The rerun differs because an unpinned layer moved, and without the layers recorded there is no way to say which one, so the difference cannot be explained, bounded or fixed — only hoped about.
How to build it
Most important first.
- Decide which reproducibility you need before you promise one. Regulatory replay may need bit-for-bit; a research comparison needs statistical; both need the inputs pinned.
- Pin everything that is an input: the commit, immutable dataset and feature versions, a resolved config, a lock file for dependencies, the container image digest for the runtime, and the hardware class (Experiment Tracking).
- For bit-for-bit: enable deterministic kernels, fix the worker and thread counts, fix the data sharding, and accept the slowdown. Record that this was done, because a rerun without it will not match.
- For statistical: report the variance across several seeds on the same inputs, and define "reproduced" as landing inside that band (Metric Uncertainty).
- Ship the artifact with its preprocessing so the serving path does not have to reproduce it from source (Preprocessing Lives in the Artifact).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For bit-for-bit: whether the artifact digest of the rerun equals the original. Binary, and the only number that answers the auditor.
- For statistical: the metric difference between the rerun and the original against the across-seed standard deviation on the same inputs. Inside the band is reproduced; outside is a changed input you have not found.
- Do not report "the metric was close". Close relative to what band, produced by which changes, is the whole question.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every input to the run — code, data, features, config, dependencies, runtime, hardware class, seeds — is recorded as an identifier that resolves to the same thing, and the rerun uses those identifiers rather than "the current" anything.
- The rerun executes the same floating-point operations in the same order as the original, or the acceptable deviation from that has been measured across seeds and stated.
- The libraries in the lock file interpret the seed and the config the same way they did at training time — no default changed in a version that was not pinned.
- Offline: a reproduction test in CI — rerun a small pinned job from its record and compare digests (bit-for-bit) or metrics within the band (statistical) — so the ability to reproduce is exercised before it is needed (Training Smoke Tests).
- Online: for the production artifact, a recorded statement of which reproducibility level it has and the band, attached to its registry entry.
- Over time: rerun a sample of historical records quarterly; a growing fraction that no longer reproduces means an input has drifted out from under the identifiers.
What can go wrong
- Deterministic kernels are enabled for the audit rerun but were not enabled for the original, so the two differ anyway; determinism has to be on at the original run to be usable later.
- The lock file is recorded but the base image was rebuilt from a floating tag and carries a different CUDA; the Python dependencies match and the numbers do not (Dependency Pinning).
- The rerun is on different hardware — a newer GPU with different reduction behaviour — and the team concludes the code is non-deterministic when the environment is.
- Statistical reproduction is declared with a band estimated from two seeds, which is not a band.
- Deterministic kernels and fixed worker counts cost training speed, sometimes substantially, and fix the run to one hardware configuration.
- Pinning every dependency and the runtime image is a maintenance cost — upgrades become deliberate events with a reproduction test — that a fast-moving team feels every week.
- Statistical reproduction needs several seeds per configuration, which multiplies training cost by the number of seeds at exactly the moment budgets are tightest.
- "We set the seed, so it is reproducible." The seed fixes the random draws. It does not fix the kernels, the reduction order, the data sharding, or the library versions that consume it. It is necessary and it is not sufficient.
- "The numbers differ in the fourth decimal, so it is fine." In a classifier with a threshold, a fourth-decimal change in scores flips the applicants nearest the threshold, which is the group the auditor is asking about.
- "Non-determinism is a bug in the framework." Non-associative floating-point reduction is a property of the arithmetic; the framework chooses speed by default and offers determinism at a cost. The bug is not recording which choice was made.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICA gradient-boosted tree trained single-threaded on CPU with a seed is usually bit-for-bit reproducible with little effort; deep networks on GPUs are the case where kernels, reduction order and worker sharding make the seed insufficient, and the rest of this lesson is mostly about them.
- DOMAIN-SPECIFICCredit, insurance and medical models can face a regulatory demand for exact replay, where bit-for-bit is the requirement and its cost is not optional; a recommender needs statistical reproducibility for honest comparisons and nothing more.
- CONTESTEDA serious position holds that bit-for-bit reproducibility is a distraction: it slows training, ties runs to specific hardware, and what actually matters is that a rerun lands inside the seed-variance band, since any conclusion that depends on bits beyond that band is not a robust conclusion. The counter is that the band is only known if several seeds were run, and that in regulated decisions "the same model" has a legal meaning that a band does not satisfy.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Programming languages and runtime internals — IEEE 754 floating-point semantics, non-associativity, and how a runtime or compiler is permitted to reorder reductions are the foundation this lesson stands on and does not teach.