Drift Explorer
Twelve weeks of traffic scored by a model that is never retrained. Pick what changes in the world — or in the pipeline — and read the monitors the way you would at 09:00 on a Monday: the input signals arrive immediately, the quality signal arrives when the labels do, and the question is whether to retrain.
Two cases on this page are the reason it exists. In data drift the model handles, PSI on the main feature clears the alert line and stays there, the prediction mean moves — and accuracy does not fall, because the true relationship is still the one the model has. A rule that says “PSI above 0.2 means retrain” retrains this model for nothing. In concept drift the opposite happens: no input distribution moves, nothing on the feature dashboard fires, and quality falls — but you can only see it once the labels arrive, weeks after the change. Drift is a signal to look. It is not the verdict.
What changes
Six generating processes for the same model. The label delay decides how late the quality signal is.
From week 5 the main feature shifts by more than a standard deviation. PSI fires. Accuracy does not move, because the true relationship is still the one the model learned.
The monitors
Dotted vertical line: the week the generating process changed (if it did). Hatching: weeks whose labels have not arrived.
Should we retrain?
Commit before you open the answer. The point of the exercise is the moment you would have been wrong.
How to read this page honestly
What the model is, and what it deliberately refuses to be.
- SIMULATEDA fixed logistic model with hand-chosen weights scores synthetic traffic for twelve weeks. Every reading — PSI, prediction mean, null rate, accuracy — is computed from the rows the scenario's generating process produced. The signals do not know which scenario made them; only the verdict, revealed after you commit, uses the scenario's identity.
- SIMULATEDGround-truth delay is enforced, not decorated. Accuracy for week *w* is null until week *w* + delay, for every scenario. The lab does not get to peek, and neither would you.
- SIMPLIFIEDThree features, four hundred rows a week, one threshold. Real monitoring works per slice and per feature with multiple reference windows, and PSI is one of several distances. The shapes — drift without damage, damage without drift, a bug that looks like drift — are the part that transfers.
- CONTESTEDWhether to retrain on a schedule regardless of signals is genuinely argued. The strongest case for the schedule is that it removes the judgement call this page asks you to make; the strongest case against is every scenario here where the retrain makes the next model worse.