Monitoring & Drift
Data, feature, prediction and concept drift taught separately; drift that is not failure; ground truth that arrives weeks late; and performance decay diagnosed rather than assumed.
A model needs everything a service needs, plus three distributions a service does not have: features in, predictions out, and outcomes back. Four layers, each with an owner, each catching a different failure.
The input distribution changed. A distance metric between the training reference and this week's traffic says so on the day; whether it matters depends on where the inputs moved to, and that needs the outcomes.
One feature's distribution moved. Before it is drift it might be a bug: a null-rate spike, a unit change, a renamed category. Diagnose the pipeline first, because retraining on a broken feature teaches the model that broken is normal.
The output distribution moved. It is the earliest model-level signal, needs no labels, and is the one that catches train/serve skew on rollout day — because the model reacts to its inputs immediately and to the truth never.
The relationship between features and outcome changed. The inputs did not move, so no input monitor fires; the scores did not move, so no prediction monitor fires. Only the outcomes reveal it, and they arrive late.
A distribution can change legitimately and the model can handle it. "Drift means retrain" retrains a working model on the strength of an input metric, costs a training run and a rollout, and answers a question the metric never asked.
The outcome arrives weeks or months after the prediction. Every quality number on the dashboard is about the past; the architecture has to say how far past, join outcomes back by id, and use proxies honestly in the meantime.
Quality over time went down. Five different causes produce that chart, only one of them is fixed by retraining, and two are made worse by it. Diagnose in order — bug, product change, feedback loop, data drift, concept drift — before touching the model.