Random Seeds
A seed fixes which random draws the code makes — the split, the initial weights, the shuffle, the dropout masks. Each is a different seed with a different effect, and the variance across them is a number a good comparison reports.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a random seed actually control, which seeds matter for which decisions, and why should a metric be reported across several of them?
Two engineers each ran a candidate churn model and one beat the other by a small margin. The lead asks whether the margin is real. "We used the same seed," they say. Rerunning both with a different seed reverses the order.
Fix one seed, use it for every run, compare metrics directly. With the seed fixed the only difference between runs is the thing being tested.
The fixed seed fixes one particular split and one particular initialisation. A candidate that happens to suit that split by chance wins; on the next seed it loses. The comparison measured the seed as much as the configuration.
- The fixed seed fixes one particular split and one particular initialisation. A candidate that happens to suit that split by chance wins; on the next seed it loses. The comparison measured the seed as much as the configuration.
- Because everyone uses the same seed, the whole team tunes against the same validation fold for months, and the fold's peculiarities become the team's conventional wisdom (Cross-Validation).
- A framework's default seeding differs by component: the data loader's workers reseed from the process id, so "the same seed" gave a different shuffle on a machine with more cores, and the reproduction failed for a reason nobody looked for (Reproducibility).
- The margin that decided the shipping decision was smaller than the across-seed standard deviation, which nobody had computed, and the winner in production is indistinguishable from the loser.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts churn; the seed question's target is whether the difference between two configurations is larger than the difference between two seeds of the same configuration.
- A single-seed metric is one draw from a distribution; the decision needs the distribution's width.
- The same dataset, feature version and config for both candidates; the only differences are the hyperparameters under comparison and — implicitly — the random draws each run makes.
- The random draws: which rows land in the validation fold, the initial weights, the order examples are visited, the dropout masks, and the negative sampling if any. Each is drawn from a generator that a seed controls.
- A budget for a handful of extra runs, not for dozens.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A pseudo-random generator produces a deterministic sequence from a seed. Every place the code draws — splitting, initialisation, shuffling, dropout, sampling, augmentation — consumes from some generator, and the seed of that generator decides the draws. Different libraries keep different generators; a single "set the seed" call reaches only the ones it knows about.
- Each draw affects a different thing. The split seed decides which examples are evaluated on, and so which metric is computed — a large effect on small data. The initialisation seed decides where optimisation starts, and so which local optimum it lands near — a large effect on deep networks, none on a convex model. The shuffle seed decides the batch composition; the dropout seed the masks.
- The metric on one seed is one sample from the distribution of metrics over seeds. Reporting the mean and spread over several seeds estimates that distribution; comparing two single-seed numbers estimates nothing about whether the difference is real (Metric Uncertainty).
Which seed controls what
The word "seed" hides four or five different decisions. Each draws from a generator and each affects a different part of the run; the effect sizes differ by model family and data size. Naming them separately is what makes it possible to vary one and hold the rest.
The split seed is special: it changes what is measured, not how the model is trained. Two candidates on different split seeds are evaluated on different examples and their metrics are not comparable at all.
| Seed | Controls | Large effect when | Vary it to measure |
|---|---|---|---|
| split_seed | Which rows are in train / validation / test | Data is small or has rare classes | Fold variance — how much the metric depends on which examples were held out |
| init_seed | Initial weights | Deep networks; non-convex objectives | Optimisation variance — sensitivity to the starting point |
| shuffle_seed | Order of examples per epoch; batch composition | Small batches; curriculum-sensitive training | Batch-order variance |
| dropout_seed | Dropout and augmentation masks | Heavy regularisation; small data | Regularisation noise |
| sample_seed | Negative sampling; subsampling in ensembles | Recommenders; boosted trees with subsampling | Sampling variance |
Splitting on a fixed seed and training on several
A comparison between two configurations should be evaluated on the same examples — fix the split seed — and should be robust to the training draws — vary the training seeds and look at the spread. The code below keeps those separate. A single global seed does both at once and confounds them.
The spread is the unit. A gap between candidates that is smaller than the across-seed standard deviation has not been measured; it has been observed once.
1import numpy as np2 3def split_indices(n, split_seed, val_frac=0.2):4 rng = np.random.default_rng(split_seed) # only this seed decides the fold5 idx = rng.permutation(n)6 cut = int(n * (1 - val_frac))7 return idx[:cut], idx[cut:]8 9def evaluate(config, X, y, split_seed=7, train_seeds=(0, 1, 2, 3, 4)):10 tr, va = split_indices(len(X), split_seed)11 scores = []12 for s in train_seeds: # init, shuffle, dropout vary13 model = train(config, X[tr], y[tr], init_seed=s, shuffle_seed=s, dropout_seed=s)14 scores.append(metric(model, X[va], y[va]))15 return np.mean(scores), np.std(scores, ddof=1)16 17m_a, sd_a = evaluate(config_a, X, y)18m_b, sd_b = evaluate(config_b, X, y)19# A difference of m_a - m_b that is smaller than sd_a or sd_b20# has not been measured; it has been observed once.The same s is reused for init, shuffle and dropout for brevity; naming them separately is what lets a later reproduction vary one at a time when a rerun disagrees.
What must stay true for a seed to mean anything
A named seed is a promise that a specific generator was set. The promise is broken silently whenever a component keeps its own generator — data-loader worker processes, GPU random state, a third-party sampler — and seeds it from something else.
The detector is a determinism smoke test: two runs with the same named seeds must produce the same split and the same first-batch loss. When they do not, a generator is loose, and every "same seed" comparison the team has made is suspect.
Each named seed controls the generator it names, in every process and library the run uses, and nothing reseeds from the clock, the process id or the core count.
holds when Seeds are threaded explicitly into the framework, NumPy, Python, the data loader's worker initialiser and any GPU generator; a determinism test passes on the target hardware.
breaks when A library upgrade changes its seeding path; worker processes are added; a new augmentation library brings its own generator; the run moves to a machine with a different core count.
respond Find and seed the loose generator; re-mark comparisons made while it was loose as single-draw observations; do not "fix" by re-running until the numbers agree.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Best of five seeds reported | Candidate beats baseline; production matches baseline | Selection on the seed inflated the estimate | Report mean and spread; never select the seed |
| Split seed varied with training seeds | Huge spread; no candidate distinguishable | Fold variance mixed into model variance | Fix the split seed across candidates; vary only training seeds |
| Framework minor upgrade | Every historical metric shifts under the same seed | The generator consumes the seed in a new order; the split changed | Record the split hash; rerun the determinism test on every upgrade |
| Data-loader workers reseed from pid | Identical seeds, different first batch | Worker generators never received the named seed | Seed workers in the initialiser from the named shuffle seed |
How to build it
Most important first.
- Name the seeds separately —
split_seed,init_seed,shuffle_seed— and record each with the run, so a reproduction can vary one while holding the others. - Hold the split seed fixed across candidates being compared, so they are evaluated on the same examples, and vary the training seeds to measure the variance that is about the model rather than about the fold.
- Report a comparison as mean and standard deviation over several training seeds, and treat a difference smaller than the spread as no difference (Beating the Baseline).
- Rotate the split seed periodically — or use k-fold — so the team does not tune to one fold's accidents (Train / Validation / Test).
- Check which generators the framework actually seeds: data-loader workers, GPU generators and third-party samplers often need their own call.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The across-seed standard deviation of the metric for a fixed configuration. This is the unit in which a difference between candidates has to be measured.
- The gap between candidates divided by that spread. A gap under about one spread is noise for any practical purpose; the decision should be made on other grounds — cost, latency, simplicity.
- Do not measure the metric on one seed to more decimals. Precision on a single draw is not information about the distribution.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Each named seed reaches the generator it is meant to control, in every library and worker process the run uses, and no component reseeds itself from the clock or the process id.
- The candidates being compared share the split seed, so their metrics are computed on the same examples, and differ only in the training seeds and the configuration under test.
- The seed variance was estimated on the same inputs and configuration the comparison uses; a band from a different dataset size or model family does not transfer.
- Offline: a determinism smoke test — two runs with identical named seeds produce identical splits and identical first-batch losses; if they do not, a generator is unseeded (Training Smoke Tests).
- Online: for a shipped model, the registry entry states the across-seed spread at training time, so a production metric difference can be judged against it.
- Over time: when a framework version changes, rerun the determinism test; a changed split under the same seed is a silent change to every comparison.
What can go wrong
- The seed sweep is run and the best seed is reported, which is selection on the seed and inflates the estimate exactly like tuning on the test set.
- The split seed is varied along with the training seeds, and the spread measured is mostly fold variance; the model comparison is now drowned in noise that a fixed split would have removed.
- A library upgrade changes how it consumes the seed — a different draw order — and the fixed seed now produces a different split, silently changing every metric in the tracking store.
- Seeds are named and recorded but the data-loader workers reseed from time, and the shuffle is different every run regardless.
- Several seeds per configuration multiplies training cost; for large models the budget may allow only two or three, which gives a rough spread rather than a precise one.
- Naming and threading seeds through every component is plumbing that has to be maintained as libraries change how they seed.
- Reporting spreads makes small improvements disappear into the noise, which is correct and unpopular.
- "We fixed the seed, so the comparison is fair." It is a fair comparison on one draw. Whether the result generalises to other draws — other splits, other initialisations — is exactly what a single seed cannot say.
- "Seed 42 gave the best result, so we ship seed 42." The seed is not a hyperparameter to select; choosing the best of several seeds is selecting on noise and the reported metric is inflated by the selection.
- "Random seeds guarantee reproducibility." They fix the random draws. Kernels, reduction order, worker counts and library versions are not random draws, and each can change the result under a fixed seed.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICFor a convex model such as logistic regression the initialisation seed has no effect on the solution and only the split seed matters; for deep networks and boosted ensembles with subsampling the training seeds move the metric materially and the spread must be measured.
- DATA-SPECIFICOn small datasets the split seed dominates — which examples land in validation changes the metric more than any training choice — and k-fold is nearly mandatory; on very large datasets the split variance is small and training-seed variance is what remains.
- SIMPLIFIEDThe "one spread" rule of thumb for judging a gap is a heuristic for the shape of the argument; a proper comparison uses a paired test across seeds, and the numbers of seeds real budgets afford are often too few for anything but a rough band.