Feature Drift
One feature's distribution moved. Before it is drift it might be a bug: a null-rate spike, a unit change, a renamed category. Diagnose the pipeline first, because retraining on a broken feature teaches the model that broken is normal.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A single feature's distribution changed this week. Is that the world, or the pipeline — and what happens if you retrain before you know?
A churn model's monitoring shows the sessions-last-7-days feature dropping sharply on Tuesday. The model's positive rate jumped the same day. The on-call engineer's first instinct is that user behaviour changed and the model needs a refresh.
A feature changed distribution; that is drift; drift is what the monitor is for. Retrain on recent data so the model learns the new behaviour, and the positive rate will settle.
The upstream job failed for a partition and the feature store has no row for those users; serving imputes zero; the model reads zero sessions as strong churn evidence and flags them. The world did not change. A job did.
- The upstream job failed for a partition and the feature store has no row for those users; serving imputes zero; the model reads zero sessions as strong churn evidence and flags them. The world did not change. A job did.
- Retraining on this week's data teaches the model that zero sessions is common and weakly predictive, because the labels for those users — when they arrive — will show they mostly did not churn. The model learns to ignore its best feature, and the pipeline bug becomes permanent because nothing looks wrong any more.
- In the Drift Explorer's feature-bug scenario, the null rate and the prediction mean move on the same day, with no label delay at all; the quality drop follows when labels arrive. The input signal was there from the first hour. The retrain hides it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict thirty-day churn; the label arrives thirty days later. The feature in question is an activity count that the model weights heavily.
- The decision is who goes on the retention call list. A feature arriving as zero for a large share of users puts the wrong people on it, immediately.
- The feature is computed by a nightly aggregation job over an event stream and written to a serving store. The serving path reads it by user id and imputes zero when the key is missing.
- The prediction log records the served feature value — the imputed zero, not the null — unless the logging is done upstream of imputation.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Feature drift is a change in the distribution of one feature, as served. It has two families of cause: the world (users behave differently) and the pipeline (the computation, the join, the null policy or the unit changed). Both produce the same distance metric; the difference is in the shape of the change and in what else moved.
- Pipeline causes have signatures. A null-rate or a spike at the imputation value is a missing join. A step change in mean by a constant factor is a unit or a window change. A new category value with the old ones vanishing is a rename. A change on one day at a job boundary, not spread over weeks, is a deploy.
- The prediction distribution moves the same day because the model responds to its inputs immediately; this makes prediction drift the confirmation, and the feature monitor the pointer (Prediction Drift).
Bug signatures
The world changes gradually, across users, over weeks. A pipeline changes on a deploy, at a job boundary, for a partition. The shape of the change is the first diagnostic, and it is available before any distance metric is computed.
The table is the on-call checklist. Every row is a change the model cannot distinguish from behaviour, and every one has a response that is not retraining.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Upstream aggregation job fails for a partition | Null rate or share at the imputation value jumps on one day; prediction positive rate moves the same day | Feature store has no row for affected keys; serving imputes the default | Rerun the job; serve the previous value rather than the default until it lands; do not retrain |
| Window or unit changed in a refactor | Mean moves by a near-constant factor at a deploy boundary; distribution shape otherwise preserved | Seven-day count became thirty-day, or seconds became milliseconds | Revert or version the feature; the trained model expects the old definition (Train / Serve Skew) |
| Category renamed upstream | A category vanishes and an unknown value appears at the same rate | The encoder maps the unknown to the "other" bucket | Map the new name in the encoder; the model has never seen "other" at this rate |
| Source table late | Feature values freeze at yesterday's for everyone | Freshness failure; the serving store was not updated | Freshness alert on the store; the distance metric will not fire because yesterday looks like today (Feature Freshness) |
| Genuine behaviour change | Gradual shift across weeks, across users, consistent with product analytics | The world | Slice quality by the shifted segment as labels arrive; retrain if the slice shows the model is wrong |
The retrain that makes the bug permanent
In the explorer's feature-bug scenario the main feature arrives as null for a large share of rows from week 7 and is imputed to zero. The model scores zero as if it were a real, low value; the prediction mean shifts the same week; accuracy falls when the labels arrive, because the users' real behaviour did not change and their real feature value would have scored them correctly.
A retrain on the weeks with the bug sees zero for those rows and labels drawn from their true behaviour. The cleanest fit is to trust the feature less. The new model's accuracy on the broken data is better than the old model's — and its accuracy on correct data is worse, and the null spike now looks normal to every monitor because the reference was refreshed.
The retrained model's validation metric on the recent weeks is higher than the old model's on the same weeks.
Positive rate settles. Retention-call precision, when the labels arrive, is lower than before the incident for users whose feature arrives correctly — the majority.
- 1The retrain learned to discount a feature that was correct for most users, because a large minority had it replaced with zero; the validation set shared the bug and rewarded the discount.
- 2The pipeline was never fixed, because after the retrain nothing looked wrong: the null spike became the reference and the positive rate returned to normal.
- 3The "recovery" was measured against broken data; the cost was paid on the correct data, which is most of it.
What the feature monitor is really checking
The model assumes the feature it receives is the feature it was trained on: the same definition, the same window, the same null policy, arriving for every user it is asked about. The feature monitor checks that assumption. Distribution distance is one of its instruments; null rate, default share and job completeness are the others, and they are the ones that catch bugs.
The response to a breach is to find out which side changed — the world or the pipeline — before deciding anything about the model.
Each feature reaches the model with the same definition, unit, window and null policy it had in training, and arrives at all for the users being scored.
holds when The pipeline's completeness and schema tests pass on every run; the null rate and default share stay at the training-set baseline; the feature definition is versioned with the model.
breaks when A job fails for a partition; a refactor changes a window or a unit; a source renames a category; a serving cache falls behind and serves stale or default values.
respond Correlate with deploys and job runs; fix or revert the pipeline; serve the previous value rather than the default meanwhile; retrain only if the change is confirmed to be the world and the quality slice shows it matters.
1SELECT date_trunc('day', served_at) AS day,2 avg(CASE WHEN raw_sessions_7d IS NULL THEN 1 ELSE 0 END) AS null_rate,3 avg(CASE WHEN raw_sessions_7d = 0 THEN 1 ELSE 0 END) AS true_zero_rate,4 avg(CASE WHEN score >= 0.5 THEN 1 ELSE 0 END) AS positive_rate5FROM prediction_log6WHERE served_at > now() - interval '14 days'7GROUP BY 1 ORDER BY 1;Two columns, not one. If the log only had the imputed value, null_rate and true_zero_rate would be a single number and the on-call engineer could not tell a job failure from a quiet week.
How to build it
Most important first.
- Monitor null rate and the share at the imputation value separately from the distribution distance, per feature; a null spike is a bug signature and deserves its own alert.
- Log the feature before imputation, so the log can distinguish "arrived as zero" from "arrived as missing and became zero".
- Run the data and feature tests on every batch of the feature pipeline — schema, null rate, range, row count against the previous run — so a broken partition fails the job rather than serving zeros (Data & Feature Tests).
- Correlate the change with deploys and job runs before any modelling explanation is entertained; the answer is usually a change someone made (ML Incident Debugging).
- Fix the pipeline and let the model recover on its own. Retraining is the response to the world changing, not to a job failing (Drift Is Not Failure).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-feature null rate and share at the default value, daily, against the stable-period baseline. This is the number that says "bug".
- Per-feature distribution distance for the non-null values, which separates "the values that arrived changed" from "fewer values arrived".
- Prediction positive rate on the same day, as confirmation that the feature change reached the model.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The feature as served is computed by the same definition, over the same window, with the same null policy, as the feature the model was trained on (Train / Serve Skew).
- The feature pipeline's completeness — row count and key coverage per run — is tested and a partial run fails rather than serving defaults.
- The prediction log records the feature before imputation, so a null can be told from a real zero after the fact.
- Offline: for each feature, record its null rate and its share at the default in the training set; a production reading far above either is a pipeline signal before it is a drift signal.
- Online: a daily comparison of feature-store key coverage against the active user set, so a missing partition is found before the model reads it.
- Over time: when a feature alert is explained as "the world", check that the explanation predicts what else should have moved — a real drop in sessions should show in the product analytics too.
What can go wrong
- The null rate is stable because the serving path imputes before logging, and the monitor sees a distribution with a spike at zero and calls it behaviour.
- The pipeline is fixed, but a retrain already ran on the broken week and the new model has down-weighted the feature; the fix restores the input and not the model.
- The feature is upstream of the model team, the alert goes to them, and the pipeline team hears about it from the retention team a week later.
- Logging pre-imputation features doubles the logged fields for every feature with a default.
- Failing a pipeline job on a completeness test means serving stale features instead of default ones, which is its own degradation to design for (Serving Fallbacks).
- Correlating with deploys requires the model team to see the pipeline team's change log, which is an organisational request.
- "The feature drifted, so user behaviour changed." A null-rate spike on a job boundary is a bug, and calling it drift misdirects the response from the pipeline to the model.
- "The retrain fixed it — accuracy recovered." The retrain taught the model to work around a broken input. The input is still broken, and the model is now worse on every user whose feature does arrive.
- "Impute zero and move on; the model is robust to missing values." The model learned that zero sessions means churn. Imputing zero for a missing row tells it the user has churned.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a pipeline failure and a world change produce the same distance metric, and that the signatures differ, holds for any feature pipeline feeding any model.
- FRAMEWORK-SPECIFICWhether a missing key is served as null, as a default, or as an error depends on the feature store or serving library, and the safest configuration — fail loudly — is rarely the default; check what yours does before trusting the null-rate monitor.
- SIMULATEDThe feature-bug scenario — a null rate near 0.4 imputed to zero from week 7, the prediction mean moving the same week — is the Drift Explorer's synthetic model on generated traffic, for the shape of the argument, not a measurement.
Where the depth lives
This domain teaches the model and hands the rest off by name.