Data Drift
The input distribution changed. A distance metric between the training reference and this week's traffic says so on the day; whether it matters depends on where the inputs moved to, and that needs the outcomes.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The inputs the model receives no longer look like the inputs it was trained on. How do you measure that, and what does the measurement license you to conclude?
A pricing model for a delivery marketplace was trained on last year's orders. The company opened in three new cities this spring. The monitoring dashboard has been showing a distance alert on the distance-to-restaurant feature for six weeks, and the team is arguing about whether it means anything.
Compute a distance metric between the training reference and this week's traffic for every feature, alert when any exceeds a standard threshold, and retrain when it alerts. Drift is the problem monitoring exists to catch.
The distance metric fires on the new cities because their distances really are different. That is a correct measurement of a real change, and by itself it says nothing about whether the model handles longer distances correctly.
- The distance metric fires on the new cities because their distances really are different. That is a correct measurement of a real change, and by itself it says nothing about whether the model handles longer distances correctly.
- In the Drift Explorer, two scenarios move the same feature by a similar amount and get the same PSI alert. In one the model keeps its accuracy; in the other accuracy falls — three weeks later, when the labels arrive. The input monitor cannot tell them apart.
- Retraining on alert, every week, with data from the new cities that has only an hour of labels per order, produces a sequence of models each trained on a smaller and noisier set than the last.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict delivery time in minutes from order, courier and geography features; the label is the observed delivery time, available an hour after the order.
- The decision downstream is the promised delivery window shown to the customer, so systematic error in one region is a broken promise in that region.
- One example is one delivered order with its features as computed at order time. Training data covers the original cities; the new cities have longer distances and a different courier mix.
- A reference distribution per feature was frozen from the training set. Weekly serving distributions are computed from the prediction log.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Data drift is a change in P(X), the distribution of inputs, between the reference period and now. It is measured per feature with a distribution distance: Population Stability Index bins the reference at its quantiles and sums (q − p)·ln(q/p) across bins; the Kolmogorov–Smirnov statistic is the largest gap between the two cumulative distributions. Both are zero for identical distributions and never exactly zero on a finite sample.
- Whether drift hurts depends on where the traffic moved to. A model is a function fitted on a region of input space; if the traffic moves within that region, or to a region where the model's functional form happens to be right, quality holds. If it moves to a region where the true relationship bends and the model never saw enough data to learn the bend, quality falls.
- The input monitor sees P(X). Quality depends on P(Y | X) in the new region. The first is observable on the day; the second is observable when the labels arrive (Ground-Truth Delay).
What the distance metric measures
PSI takes the reference distribution, cuts it into bins that each hold about a tenth of the reference, then counts how much of the current traffic lands in each bin. Where current and reference agree the contribution is zero; where a bin has emptied or filled the contribution grows with both the difference and its log ratio. On a finite sample it is never zero, and its noise floor depends on the sample size and the feature.
The number is a statement about the inputs. It is computed before any label exists, which is why it is the earliest signal and why it cannot say whether the model is wrong.
1import math2 3def psi(reference, current, bins=10, eps=1e-4):4 ref = sorted(reference)5 edges = [ref[min(len(ref) - 1, (b * len(ref)) // bins)] for b in range(1, bins)]6 def bin_of(v):7 i = 08 while i < len(edges) and v >= edges[i]:9 i += 110 return i11 ref_counts = [0] * bins12 cur_counts = [0] * bins13 for v in reference: ref_counts[bin_of(v)] += 114 for v in current: cur_counts[bin_of(v)] += 115 total = 0.016 for b in range(bins):17 p = max(eps, ref_counts[b] / len(reference))18 q = max(eps, cur_counts[b] / len(current))19 total += (q - p) * math.log(q / p)20 return totalThe eps floor is what makes the metric finite when a bin empties. It is also what makes PSI unreliable on a categorical feature with a new value: the whole shift lands in a bin the reference never had, and the floor decides the answer.
Same alert, two outcomes
The Drift Explorer runs two scenarios that begin identically. From week 5 the main feature's mean shifts by more than a standard deviation in both. PSI clears the alert line in both, in the same week. In one, the true relationship is the one the model has and the shift stays where a linear model is right: accuracy holds. In the other, the shift carries traffic into a range above which the true relationship bends, a range the training data barely covered: accuracy falls, and the fall becomes observable three weeks after the shift.
On the day of the alert the two dashboards are indistinguishable. That is the whole lesson. The distance metric says where to look; only the outcomes say what was found.
Validation error on the original cities is unchanged; the distance-to-restaurant feature has a PSI alert since the week the new cities launched.
Overall delivery-time error looks fine for two weeks, then the new-city slice shows systematic under-prediction: long distances are a region where the model's linear-in-distance assumption never held and the training data had too few examples to show it.
- 1The traffic moved into a region of input space where the model's functional form is wrong and which the training set covered too thinly to correct — harmful data drift, invisible in the aggregate because the new cities are a small share.
- 2The aggregate error hid it; the slice by city revealed it once an hour of labels per order had accumulated into enough orders per city.
- 3A harmless version of the same alert was equally possible: had the new cities' distances stayed in the range the model handles, the same PSI would have meant nothing.
The assumption a distance alert is checking
The model was fitted on a region of input space, and its weights are a good approximation of the true relationship on that region and an unknown one elsewhere. The drift monitor is the check that traffic still lives where the model was fitted; it is not the check that the model is right, because the model can be right outside its region by luck of functional form.
The response to a breach is therefore a slice, not a retrain. The retrain is what follows if the slice shows a problem, and it needs data from the new region to fix anything.
The serving-time input distribution lies within the region of input space the training data covered densely enough for the model's functional form to be a good approximation.
holds when Per-feature distance to the training reference stays within the stable-period noise floor, or moves into a region where the model's form matches the true relationship — which is only knowable from outcomes.
breaks when A product expansion, a new acquisition channel or a seasonal shift carries traffic into a range the training data covered thinly; the model extrapolates and the extrapolation is wrong.
respond Identify the region and its share; wait for or accelerate the slice; retrain with labelled data from the new region only if the slice shows the model is wrong there.
PSI on distance_to_restaurant crossed the threshold. Retraining job triggered on last four weeks of data.
PSI on distance_to_restaurant crossed the threshold; the shift is the three new cities, which are a small share of traffic. Pipeline checks are clean. Quality on the new-city slice will be readable once enough orders have labels; predicted delivery windows in those cities are widened until then.
The first retrains on a signal that cannot distinguish harmless from harmful drift, on data whose labels are mostly not in yet. The second names the region, rules out the bug, and puts a mitigation in place while the only decisive number is still arriving.
How to build it
Most important first.
- Measure per feature against a reference frozen from the training set, with the reference's own week-to-week variance established so the threshold is set above noise (Model Monitoring).
- When a feature alerts, first ask the pipeline question — is this the world or a bug? A null-rate spike, a unit change or a schema change is a bug (Feature Drift).
- If it is the world, slice quality by the drifted region as soon as labels exist: delivery-time error in the new cities specifically, not overall. That is the question the alert raised.
- Retrain when the slice shows the model is wrong in the new region, with enough labelled data from that region to learn it — not on the alert, and not before (Drift Is Not Failure, Retraining as a Decision).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature PSI or KS against the training reference, weekly, with the stable-period variance shown alongside so the reader can see what noise looks like.
- Quality on the drifted slice once labels arrive. This is the number that decides whether the drift mattered; the distance metric only says where to look.
- The share of traffic in the drifted region — a shift affecting a tenth of orders has a tenth of the business impact of the same shift on all of them.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The training reference distribution still describes the population the model was meant to serve; a change in reference is a modelling decision, not a monitoring convenience.
- The distance metric is being computed on the feature as served, from the prediction log, not on a re-computed batch version of the feature.
- Quality can be sliced by the drifted region when labels arrive, which requires the region to be identifiable in the joined outcome data.
- Offline: compute each feature's distance metric between the training set and a held-out period from the same era, to learn the metric's noise floor before setting an alert.
- Online: when a feature alerts, produce the slice — the drifted region's share of traffic and its predicted distribution — the same day, before any retraining conversation.
- Over time: quality on the drifted slice as labels arrive, compared with quality on the unchanged slice; a gap that opens confirms the drift mattered, a gap that does not means it did not.
What can go wrong
- The reference is refreshed to "last month" so the alert stops firing, and slow drift becomes invisible because the reference walks with it.
- PSI is computed on a categorical feature with a new category that was absent from the reference; the empty-bin floor dominates and the metric is meaningless.
- The drift is real, the model is wrong in the new region, and the alert had been muted for six weeks because it fired continuously (Alert Fatigue: The Page Nobody Reads).
- Per-feature distance monitors are one threshold per feature to tune, and the untuned defaults produce a red dashboard.
- Waiting for the slice means waiting for labels; in the meantime the model is either serving a wrong prediction in the new region or the team has acted on incomplete information.
- Freezing the reference means the monitor fires on legitimate long-term change, which is the design, and also what makes it annoying.
- "Drift means retrain." It means the inputs changed. The explorer's harmless scenario is the counterexample: PSI far above the alert line and no quality loss at all. Retraining there costs a training run and a rollout risk for nothing.
- "PSI is below the threshold, so the model is fine." PSI measures P(X). Concept drift leaves P(X) untouched and breaks the model anyway.
- "Use a standard PSI threshold." The conventional cut-offs come from credit scoring on particular features; a feature with heavy tails or few distinct values has a different noise floor, and the threshold should be set from that feature's stable-period variance.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat P(X) can change without P(Y | X) changing, and that an input monitor sees only the first, holds for any model family and any task.
- DATA-SPECIFICPSI and KS are for numeric or low-cardinality features; on high-cardinality categoricals, text or images the practical monitors are embedding-space distances or a classifier trained to distinguish reference from current traffic, and their thresholds have no conventional defaults at all.
- SIMULATEDThe two data-drift scenarios and the PSI values they produce come from the Drift Explorer's synthetic model on generated traffic, chosen to show the shape of the argument; the alert constants are the explorer's, not a recommendation.
Where the depth lives
This domain teaches the model and hands the rest off by name.