Data Leakage
The deepest module. Target, temporal, entity, preprocessing, feature and evaluation leakage: every way information from the answer reaches the model, and why each makes offline metrics lie.
Leakage is information from the answer reaching the model during training through a route that will not exist at prediction time. The offline metric improves; the product does not.
A feature that is derived from, caused by, or written by the same process as the label. It looks like a column; it is the answer.
Information from after the prediction time reaches the features: a future timestamp, a window that crosses the snapshot, a random split of time-ordered data.
The same user, patient or device appears on both sides of the split. The model memorises the entity, the evaluation rewards it, and production is full of strangers.
A scaler, imputer, encoder or feature selector fitted on the full dataset before the split has seen the validation rows. The order of operations is the leak.
The data is clean and the pipeline is ordered correctly. The leak is the engineer: tuning on the test set, peeking repeatedly, picking the best of many runs on one holdout.
A checklist run on every feature before the offline number is believed: when is it computed, from what, is it available at prediction time, does it correlate suspiciously, is it near-perfect on a subgroup.