Sampling Strategies
Random, stratified, temporal and group-based sampling each preserve a different property of the population. Which property matters depends on what the model will meet in production.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The full dataset is too large or too skewed to use as is. Which subset can be trained on without teaching the model a population that does not exist?
A payments company logs a billion card authorisations a year. Training on all of them is a week of compute. The team wants a sample small enough to iterate on in an hour and still representative of what the fraud model will score tomorrow.
Take a uniform random sample of the rows. Random is unbiased; ten million rows is plenty; anything that is true of the population is true of the sample in expectation.
Ten thousand positives is enough for the aggregate metric and nothing like enough for the slices that matter: a merchant category with a distinctive fraud pattern has a few dozen, and the model never learns it.
- Ten thousand positives is enough for the aggregate metric and nothing like enough for the slices that matter: a merchant category with a distinctive fraud pattern has a few dozen, and the model never learns it.
- The sample is uniform over the year, so the fraud rings of last month — the ones production will meet tomorrow — are one-twelfth of the positives and outvoted by patterns that no longer exist.
- The same cardholder is in the sample many times. When the team then does a random split, the cardholder's other rows leak their identity into the evaluation, and the offline number is optimistic in a way production never shows.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an authorisation will be disputed as fraud within 60 days. Positives are rare — roughly one in a thousand — and their mix changes with the merchant landscape and with the fraud rings currently active.
- The model scores every authorisation in real time, so production sees the full population, not a sample.
- One example is one authorisation with the cardholder's trailing aggregates. The same cardholder appears thousands of times a year; the same merchant appears millions of times.
- Events are timestamped and the fraud mix has a strong seasonal and weekly pattern. Labels arrive up to 60 days late.
- A naive uniform sample of ten million rows contains about ten thousand positives, spread thinly across merchants and months.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A sample is a choice of which rows to keep, and every strategy preserves some property of the full data exactly while letting others vary. Uniform random sampling preserves all proportions in expectation but with variance that is large for rare groups.
- Stratified sampling fixes the proportion of a chosen variable — the label, a segment, a month — so rare strata are represented at a decided rate. Temporal sampling weights or restricts by time so the sample resembles the near future rather than the average past. Group-based sampling keeps or drops whole entities so that the rows for one cardholder stay together.
- The strategies compose. A common production recipe is: keep every positive, downsample negatives, weight recent months more heavily, and sample by cardholder rather than by row — then correct the model's probabilities for the deliberate imbalance you introduced (Calibration).
Four strategies, four preserved properties
Each strategy is a promise about what the sample keeps fixed. The question to ask is not "which is best" but "which property does production depend on". A fraud model depends on recent positives per merchant; a manufacturing defect model on a stable process depends on the plain population rate.
Group-based sampling is the one people forget, and it is the one that later decides whether the split is honest.
| Strategy | Preserves exactly | Leaves to chance | Use when | Fails by |
|---|---|---|---|---|
| Uniform random | Nothing; all proportions in expectation | Rare strata, entity structure, recency | Balanced, stationary data with independent rows | Rare slices too thin to learn or to evaluate |
| Stratified | Proportion of the chosen variable per stratum | Everything not stratified on | Rare label or segment must be represented at a decided rate | Distorted probabilities; unstratified variables still skewed |
| Temporal | Share of rows from each period, or a recency window | Older patterns | Production resembles the near future more than the average past | Window not moved forward on retraining |
| Group-based | Whole entities kept or dropped together | Exact row count | The same entity recurs across rows | Groups of very different sizes dominate |
Sampling by cardholder, keeping every positive
The mechanism is short. Hash the entity key to decide membership, so that the same cardholder lands in the sample or out of it as a unit and the decision is reproducible without storing a list. Keep every positive regardless. Record the negative keep-rate so the probabilities can be corrected later.
The important line is the last one: the model will be trained at an inflated positive rate, and the correction to its log-odds is exactly the log of the negative sampling rate.
1import hashlib, math2 3NEG_KEEP = 0.02 # keep 2% of negative cardholders' rows4 5def keep(row):6 if row["is_fraud"]:7 return True # every positive stays8 h = hashlib.sha1(f"{row['cardholder_id']}:seed42".encode()).digest()9 return int.from_bytes(h[:8], "big") / 2**64 < NEG_KEEP10 11sample = [r for r in rows if keep(r)]12 13# The model now sees fraud at a rate ~1/NEG_KEEP times reality.14# Correct its output before using it as a probability:15def corrected_logit(model_logit):16 return model_logit + math.log(NEG_KEEP)Hashing on the entity rather than on the row means the same cardholder is always in or always out, which is what lets the split later keep them on one side. The seed in the hash string is what makes the sample versionable.
What the sample assumes about tomorrow
Whatever strategy is chosen encodes a belief about production. Recency weighting says next week looks like last quarter more than last year. Stratifying by merchant category says the category mix in production is the one in the sample. These are assumptions with the same status as any other, and they decay.
The sample is also the thing most likely to be reused unchanged across retrainings, because it was expensive to draw. That is exactly how a recency window becomes a stale window.
The label rate per stratum, the segment mix and the recency profile of the training sample match the traffic the model scores, up to the deliberate distortions that were recorded and corrected.
holds when The sample is re-drawn at each retraining with the recency window advanced, the evaluation set is a production-like uniform draw, and the negative sampling rate is applied as a calibration correction.
breaks when A new merchant category launches; a fraud ring moves the positive mix; the sample is reused for a year; the calibration correction is dropped during a serving refactor.
respond Re-draw the sample with the current window and mix. If the calibration offset moved, check the correction before suspecting the model.
How to build it
Most important first.
- State what property production traffic has that the sample must preserve — label rate, segment mix, recency, entity structure — and pick the strategy that fixes that property rather than hoping uniform sampling gets it.
- Sample entities, not rows, whenever the same entity recurs, so the sample and the later split share a grouping (Group Split).
- If you downsample negatives, record the sampling rate per stratum and reweight or recalibrate afterwards; a model trained at a 1:10 ratio outputs probabilities for a world with ten times more fraud than exists.
- Keep the sampling code and its seed with the dataset version; a sample that cannot be reproduced cannot be compared against (Reproducibility).
- Evaluate on a sample drawn to look like production — uniform, recent, all positives kept — even if training used a very different one.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Count of positives per slice that the business cares about, in the sample. This is the number that decides whether the model can learn the slice; total row count does not.
- The sample's label rate and segment mix against the most recent production week, not against the yearly average.
- After training on a downsampled set, the mean predicted probability against the true production rate — the size of the calibration correction the downsampling demands.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The property the sampling strategy preserved — label rate per stratum, recency mix, entity structure — is the property that matters for production, and production's version of that property has not moved since the sample was drawn.
- Any deliberate distortion of the label rate is known, recorded, and corrected before the model's probabilities are used as probabilities.
- The evaluation set was drawn to resemble production traffic, so the offline metric is about the population the model scores.
- Offline: compare per-slice positive counts and segment proportions between the sample and a held-back uniform draw of the latest month; a slice that is thin in the sample is a slice the metric cannot speak for.
- Online: on rollout, compare the mean predicted probability to the observed positive rate as labels arrive; a constant offset is the uncorrected sampling ratio.
- Over time: re-draw the sample on every retraining with the recency window moved forward, and diff the segment mix against the previous sample.
What can go wrong
- Negatives are downsampled and the model is served without recalibration; every threshold chosen offline is wrong by the downsampling factor and the review queue is ten times too large.
- The sample is stratified by label but not by merchant, so a large merchant contributes most of the positives and the model learns that merchant rather than fraud.
- Recency weighting is set once; a year later the "recent" window is stale and the sample is again dominated by patterns that no longer exist.
- Stratified and recency-weighted samples make the model better on the rare and recent, and worse on the common and old, which is usually the right trade for fraud and the wrong one for a stable process.
- Keeping all positives and downsampling negatives distorts the probabilities, which must then be corrected, and the correction is a second thing that can be wrong.
- Sampling by entity produces a sample whose row count you cannot choose exactly, because entities have different numbers of rows.
- "Random sampling is unbiased, so it is the safe default." Unbiased in expectation, with variance that swamps rare strata. Unbiased is not the same as representative for the slice that matters.
- "We stratified by label, so the classes are balanced and we are done." Balance between classes says nothing about balance within the positives across merchants, months or entities.
- "Downsampling negatives improved recall." It moved the decision threshold implicitly. The same recall was available from the original data at a lower threshold, with honest probabilities.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a sample preserves some properties exactly and others only in expectation is a statistical fact independent of the task or model family.
- DATA-SPECIFICThe recipe of keeping all positives and downsampling negatives is for rare-event data; on balanced data uniform sampling is fine and the calibration correction is unnecessary, while on tiny datasets the right sample is all of it.
- CONTESTEDA strong position holds that downsampling negatives is almost always a mistake with modern tooling: gradient-boosted trees and neural networks train on a hundred million rows in reasonable time, class weights achieve the same effect without discarding information, and the calibration correction downsampling demands is an avoidable source of production error. The counter-argument is iteration speed — a team that trains in an hour instead of a day runs more experiments — and that at extreme imbalance the negatives really are redundant.
Where the depth lives
This domain teaches the model and hands the rest off by name.