Stratified Split
When positives are rare, a plain random cut can leave validation with too few of them to say anything. Stratifying fixes the class ratio per set so every fold holds a known number of positives.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Are positives rare enough that a random cut could leave the validation set with too few to measure the metric, and which variable should the split hold fixed?
A SaaS company wants to detect accounts that will be flagged for abuse. Around one account in five hundred is flagged; the team's five-fold cross-validation gave wildly different scores per fold and nobody could tell which model was better.
K-fold cross-validation with random folds. Five folds average out the noise, and the mean score across folds is the estimate.
The per-fold scores vary far more than the differences between candidate models. Part of that is the metric's inherent noise at eighty positives; part is the fold-to-fold swing in positive count, which a random cut adds on top (Metric Uncertainty).
- The per-fold scores vary far more than the differences between candidate models. Part of that is the metric's inherent noise at eighty positives; part is the fold-to-fold swing in positive count, which a random cut adds on top (Metric Uncertainty).
- The fold with sixty-two positives happens to have most of them from the free-tier channel, and the model that does best on that fold is the one that learned the channel. The averaged winner is decided by which fold got which positives.
- In production, thresholds tuned on a fold whose positive rate was one in four hundred are applied to a stream at one in five hundred, and the friction rate is off by the difference (Threshold Selection).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an account will be flagged for abuse within 60 days of creation. The label is the flag, set by a trust-and-safety review.
- The decision is whether to apply friction at signup — extra verification — with a cost per false positive in lost signups and per false negative in abuse damage.
- One example is one account at creation: signup attributes and first-hour behaviour. Two hundred thousand accounts, about four hundred positives.
- Five random folds of forty thousand rows hold, on average, eighty positives each, with a standard deviation of about nine; one fold had sixty-two and another ninety-eight.
- The positive rate also differs sharply by signup channel, from near zero via enterprise sales to a few percent via one free-tier channel.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A random split assigns each row independently, so the number of positives per set is binomial and its relative variance grows as positives get rarer. With four hundred positives across five folds, the count per fold swings by more than ten percent from chance alone.
- A stratified split draws separately within each class — or within each level of a chosen variable — and cuts each stratum at the same ratio, so every set holds the same class proportion up to rounding. It removes the binomial variance in the positive count; it does not remove the metric's own noise, which depends on the count itself.
- Stratification can be on any variable whose proportion matters for the evaluation: the label, a channel, a region, or the label crossed with a segment. It fixes that variable's proportion and leaves everything else random (Sampling Strategies).
How much a random cut moves the positive count
Four hundred positives across five folds: the expected count per fold is eighty, and the binomial standard deviation is about eight. A fold at sixty-five and a fold at ninety-five are both ordinary draws, and a precision-like metric computed on sixty-five positives is a different instrument from one computed on ninety-five.
The code shows the mechanism and the fix side by side. The stratified version draws within each class and cuts each at the same ratio; the counts per fold become equal up to rounding.
1import random2from collections import Counter3 4def random_folds(rows, k, seed=0):5 rnd = random.Random(seed); idx = list(range(len(rows))); rnd.shuffle(idx)6 return [idx[i::k] for i in range(k)]7 8def stratified_folds(rows, k, key, seed=0):9 rnd = random.Random(seed)10 by_stratum = {}11 for i, r in enumerate(rows):12 by_stratum.setdefault(key(r), []).append(i)13 folds = [[] for _ in range(k)]14 for members in by_stratum.values(): # cut each stratum at the same ratio15 rnd.shuffle(members)16 for j, i in enumerate(members):17 folds[j % k].append(i)18 return folds19 20def positives_per_fold(folds, rows):21 return [sum(rows[i]["flagged"] for i in f) for f in folds]22 23# random: e.g. [62, 84, 79, 98, 77] -- binomial spread24# stratified: e.g. [80, 80, 80, 80, 80] -- fixed by construction25# stratify on label x channel: key=lambda r: (r["flagged"], r["channel"])Stratifying on label crossed with channel is the version that stops one fold being the free-tier fold. Watch the stratum sizes: a stratum smaller than k cannot be spread across every fold, and the code above just places its rows in the first few.
Stratify within, never instead of
Stratification fixes a proportion. It does nothing about entities or time, and applied naively it undoes them: stratifying on the label across the whole dataset scatters each entity's rows across folds and mixes periods. The rule is to apply the structural split first and stratify inside it.
The decision below says which constraint wins when they conflict on a small dataset. The short answer is that the structural one wins, because its failure is a wrong number and stratification's failure is a noisy one.
The dataset is small, positives are rare, and the entity or time split leaves validation with too few positives. What gives?
when Entities recur or the data drifts, and production will be new entities or a new period. The number must measure that.
cost A wide interval on the metric; candidate models may be indistinguishable, and the decision falls to cost, latency or simplicity.
when The structural split can be made at the entity or period level and the positives are spread across many entities or periods.
cost Implementation is fiddly — stratify on the label at the entity level, then assign entities — and libraries rarely do it for you.
when Data is small enough that a single holdout cannot separate candidates and there is no strong time drift.
cost k trainings; groups of very different sizes produce unequal folds; a final untouched test period is still needed.
when Only when rows are genuinely exchangeable and there is no drift — in which case there was no conflict.
cost On any other data: entity or temporal leakage, and a stable, confident, wrong number.
The positive rate the split assumed
A stratified evaluation is a statement about the class ratio it held fixed. The threshold, the expected friction rate and the per-segment metrics were all computed at that ratio. When the production ratio moves — a new abuse campaign, a new channel — those numbers describe a different population than the one arriving.
This is the same assumption class imbalance makes explicit, seen from the split's side.
The positive rate, overall and per stratified segment, in the evaluation sets is close to the rate in production traffic, so thresholds and per-segment metrics computed on them transfer.
holds when The split preserved the natural ratio; per-segment rates are re-checked at each retraining; the threshold is re-derived when the rate moves.
breaks when An abuse wave raises the rate in one channel; a new channel launches with no history; the training fold was rebalanced and the threshold tuned on it.
respond Re-derive the threshold at the current rate before questioning the model; re-stratify on the current segment mix at the next retraining.
How to build it
Most important first.
- Stratify on the label whenever positives are rare enough that the count per set matters — as a rule of thumb, whenever a set would hold fewer than a few hundred positives.
- Stratify on label crossed with the key segment when the positive rate differs strongly by segment, so that no fold is the free-tier fold.
- Report the positive count per set beside the metric, and the metric's standard error; stratification makes the count predictable, not the metric certain.
- Combine with the structural splits: stratify within a time period or within the entity assignment, never instead of them (Time-Based Split, Group Split, Choosing a Split Strategy).
- Keep the natural class ratio in every set; stratification fixes proportions, it does not rebalance them, and rebalancing belongs in training with a calibration correction, not in the split (Class Imbalance).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Positive count per set and per segment, which decides whether any comparison is possible at all.
- The per-fold spread of the metric after stratification, against before; the reduction is the variance the split was adding, and what remains is the metric's own noise.
- The mean predicted probability against the per-set positive rate, which should now match across folds.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The production positive rate is close to the rate the stratified sets hold, per segment; a stratified evaluation is a statement about that rate.
- The stratifying variable is the one that matters for the evaluation and is available for every row at split time — a label or segment missing on some rows silently forms its own stratum.
- Stratification was applied within the structural split — inside a period, inside an entity assignment — not across it.
- Offline: print the positive count per set and per stratum; compare the fold-to-fold metric spread with and without stratification.
- Online: compare the production positive rate and friction rate against the stratified set's rate and the planned queue size in the first weeks.
- Over time: re-check the per-segment positive rates at each retraining, since a segment whose rate has moved changes what the stratification held fixed.
What can go wrong
- Stratifying by label on data with recurring entities splits an entity's rows across folds, and the leakage this introduces is larger than the variance stratification removed (Entity Leakage).
- Stratifying by label crossed with too many segments produces strata with one or two rows, which cannot be split at a ratio at all and end up wherever the library puts them.
- The split is stratified, but the threshold is later tuned on a rebalanced training fold whose positive rate is not the natural one.
- Stratification constrains the split, and every constraint — label, segment, entity, time — competes with the others on a small dataset until one has to give.
- It removes one source of variance and can create a false sense that the metric is now stable when the underlying positive count is still small.
- Stratifying on many variables fragments the data into strata too small to cut, and the library's handling of those is rarely what you want.
- "Stratifying balances the classes." It preserves the natural ratio in every set. Balancing changes the ratio, which is a different operation with a calibration cost, and it does not belong in the split.
- "We stratified, so the folds are comparable and the split is done." The folds now hold the same positive count. The same user may still be in several of them, and later periods may still sit beside earlier ones.
- "Eighty positives per fold is enough because we have five folds." The folds share the same four hundred positives; five noisy estimates of the same quantity are not five times the evidence.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the per-set positive count under a random cut is binomial, and that stratifying removes that variance, is arithmetic independent of task or model.
- DATA-SPECIFICMatters when positives are rare or a segment is small; on balanced data with hundreds of thousands of rows the per-set proportions are already stable to within noise and stratification changes nothing measurable.
- CONTESTEDK-fold cross-validation versus a single stratified holdout is genuinely disputed. The case for k-fold is that on small data it uses every row for validation once and gives a spread rather than a point, and that a single holdout of eighty positives is a coin flip between candidates. The case for a single holdout is that k-fold multiplies training cost, complicates temporal and group constraints, and produces folds people quietly tune on because there is no single set to protect; many practitioners use k-fold for selection and one untouched holdout for the final number.
Where the depth lives
This domain teaches the model and hands the rest off by name.