Feature Selection
Fewer features means fewer serving dependencies, less leakage surface, and a smaller lie when the selection is done outside the training fold — which is where it is usually done.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
We have four hundred candidate features. How do we decide which ones the model should use, without the selection itself contaminating the evaluation or leaving us with columns production cannot supply?
A lending team's feature table has grown to four hundred columns over three years. Training takes hours, the serving service depends on eleven upstream tables, and two of the top features were quietly deprecated last quarter. Someone ran a correlation ranking against the label over the whole table and proposed keeping the top forty.
Rank every column by its correlation with the label over the full dataset, or train one big model on everything and keep the columns with the highest importance. Drop the rest. The evaluation improves or holds steady, training is faster, and the serving service has fewer dependencies.
The ranking was computed on the whole dataset, including the rows that become the validation set. Features that correlate with the label by chance in those rows are selected, and the validation metric rewards them because it was part of the selection. The estimate is optimistic and the size of the optimism is unknown (Preprocessing Leakage).
- The ranking was computed on the whole dataset, including the rows that become the validation set. Features that correlate with the label by chance in those rows are selected, and the validation metric rewards them because it was part of the selection. The estimate is optimistic and the size of the optimism is unknown (Preprocessing Leakage).
- One of the top forty is
days_since_account_closed, which is null for everyone except people whose account was closed for delinquency. It is a leaked label wearing a feature name, and the ranking loved it (Target Leakage). - Two kept features come from a table that is refreshed weekly; the serving service reads it live and gets a stale value for a week after every applicant change. The training data used the eventually-correct value.
- Six months later an upstream team drops one of the eleven tables. The three features that came from it start arriving as null and the model treats "null" as it learned to — as a signal — for every applicant.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a loan applicant will miss a payment in the first twelve months. The label is the first delinquency event, observed up to a year after the decision.
- The decision is approve, decline, or refer to an underwriter. The model has to run at application time from the features the application service can actually fetch.
- One example is one application with the applicant's bureau data, declared income, and aggregates over any prior accounts, as of the application timestamp.
- Four hundred columns produced by many people over three years, with overlapping definitions — six different "income" variants, three "months since last delinquency" with different null policies.
- Roughly thirty features come from a table refreshed weekly; the rest from nightly or on-demand sources. The serving service must call every upstream that any kept feature needs (Feature Freshness).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Filter methods score each feature against the label independently — correlation, mutual information, a univariate test — and keep the top k. They are cheap and blind to interactions: a feature that matters only in combination scores low, and a leaked feature scores highest of all.
- Wrapper methods ask the model: add or remove features and re-evaluate. They see interactions and cost a training run per step; with a validation set reused across hundreds of steps they overfit the validation set itself (Never Tune on the Test Set).
- Embedded methods select during training — an L1 penalty driving coefficients to zero (Regularized Linear Models), or a tree ensemble that never splits on a feature. They are cheap and specific to the model family, and a tree that never used a feature says nothing about whether a linear model would.
- All three are fitted procedures. Anything fitted on data that includes the validation rows has seen the validation labels, and the validation metric after selection is no longer an estimate of anything.
Three ways to select, and what each cannot see
Filter, wrapper and embedded selection differ in who is asked. A filter asks each feature on its own — cheap, and blind to any feature that only matters with another one. A wrapper asks the model repeatedly, which sees interactions and spends a training run per question. An embedded method lets the model decide while it trains, which is free and applies only to that model.
The one thing all three agree on is that a leaked feature is the best feature. Whatever ranks features by how well they predict the label will put a proxy for the label at the top, and selection therefore concentrates leakage rather than removing it. The audit has to come after the ranking, not instead of it.
What are you willing to spend, and what must the selection see?
when Hundreds of candidates, a first pass, or a model family with no embedded selection; interactions are not expected to matter much.
cost Cannot see interactions; ranks leaked proxies highest; must be re-fitted inside every fold to avoid contaminating the validation set.
when Tens of candidates and a cheap model, where interactions matter and a training run per step is affordable.
cost One training run per step; hundreds of evaluations against one validation set overfit it, so the result needs a further untouched holdout.
when The model family supports it and the selected set will be served with that family; you want selection and training to be one procedure.
cost The verdict is specific to the model and to the correlated siblings present; a redundant feature is dropped arbitrarily rather than on merit.
Selection is a fitted step and belongs inside the fold
Ranking features against the label is fitting: it reads the labels and produces parameters (the kept set). If the labels it read include the validation rows, then the validation metric is evaluating a pipeline that has already seen the answers. With four hundred candidates, some correlate with a few thousand validation labels by chance, and the ranking selects exactly those.
The fix is the same as for a normaliser or a target encoder: fit on the training part of the fold, apply to the held-out part, and repeat per fold. The selected set may differ between folds, and that instability is information — it says the features are near-interchangeable and the ranking is noise near the cut-off.
looks like A config file listing forty feature names, produced by a one-off notebook that ranked all four hundred columns by correlation with the label over every row.
why it leaks The ranking read the validation labels. Features that correlate with those labels by chance were selected, so the validation set is no longer independent of the pipeline it is scoring.
fix Run the selection inside each training fold; evaluate on the fold's held-out part; keep the final holdout untouched by any ranking.
1def evaluate_with_selection(X, y, folds, k, fit_model, score):2 results = []3 for train_idx, val_idx in folds:4 X_tr, y_tr = X[train_idx], y[train_idx]5 # the ranking reads only training labels6 ranking = univariate_score(X_tr, y_tr) # one number per column7 keep = top_k(ranking, k) # the "parameters" of this step8 model = fit_model(X_tr[:, keep], y_tr)9 results.append(score(model, X[val_idx][:, keep], y[val_idx]))10 return mean(results), std(results)11 12# WRONG: keep = top_k(univariate_score(X, y), k) before the loop13# -- the validation rows were read by the ranking, so every fold is optimisticThe kept set is recomputed per fold. If it changes a lot between folds, the ranking near the cut-off is noise, and the honest statement is "any of these forty-ish features" rather than "these forty".
Every kept feature is a dependency
Offline, a feature is a column. In production it is a call to a source, a freshness guarantee, a null policy, a monitor, and a team who can deprecate it. Selection is therefore also a serving decision: forty features from three sources are a different system from forty features from eleven sources, even at the same validation score.
This is why the serving filter runs first. Removing what production cannot supply is not a statistical question and reads no labels; doing it before the ranking means the ranking is over features the model could actually receive.
- 1Serving filter
Drop features the serving path cannot supply at the needed freshness, deprecated sources, and features whose serving null rate differs from training. Reads no labels.
fails by Skipped because the notebook already has all four hundred columns loaded and it is easier to rank them all.
- 2Leakage screen
Flag features with implausible strength or a null pattern tied to the outcome; check what time each is computed at.
fails by Run after the ranking on the top forty only, so the leaked proxy that ranked first is kept because it "clearly works".
- 3Nested statistical selection
Inside each fold, rank on training labels, keep k, fit, evaluate on held-out.
fails by Collapsed to a single ranking on the full table for speed; validation becomes optimistic by an unknown amount.
- 4Record the procedure, not the list
The retrain pipeline runs the same selection; the artifact records which features it kept and why.
fails by The list is copied into a config file and outlives every reason it was chosen.
Every selected feature is present at serving time with the null rate, freshness and definition the selection and the training data reflected.
holds when Each feature's source is owned, monitored, and part of the serving contract; deprecations upstream are announced to the model's owners; the retrain pipeline re-runs selection rather than reading a frozen list.
breaks when An upstream table is dropped and the serving service substitutes a default; a weekly-refreshed source is read live; a feature's definition is changed by its owner without the model being retrained.
respond Treat a missing or moved feature as an incident, not as drift. Restore the source or retrain without the feature; do not let the model keep reading the default as a signal.
How to build it
Most important first.
- Do the selection inside the training fold. In cross-validation that means inside each fold, selecting on the training part and evaluating on the held-out part; the selection is part of the pipeline being evaluated, not a step before it (Cross-Validation).
- Select for serving reality first. Remove any feature the serving path cannot supply at the required freshness, any feature from a deprecated source, and any feature whose null rate at serving time differs from training — before any statistical ranking.
- Run a leakage audit on whatever the ranking puts at the top. A feature that is suspiciously strong is more likely leaked than brilliant (The Leakage Audit).
- Prefer fewer, legible features over the best forty of four hundred. Every feature kept is an upstream dependency, a monitor, and a train/serve skew opportunity for the life of the model.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Held-out metric where the selection was fitted inside the fold, compared with the same metric on the full feature set. If the gap is small, the smaller set wins on cost alone.
- The count of upstream dependencies and the worst-case feature freshness after selection — that is what the serving team is buying.
- Do not measure the validation metric after a selection that saw the validation rows. It is a number about the selection procedure, not the model.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every selected feature can be computed at serving time from the sources the serving path actually calls, at the freshness the training data reflected.
- The null rate and the distribution of each selected feature at serving time match what the selection saw; a feature selected because it was rarely null is a different feature when it is often null.
- The selection procedure that produced the feature set is part of the retraining pipeline, so a retrain re-selects rather than inheriting a list frozen in a config file.
- Offline: nested cross-validation where the selection runs inside each fold; compare against a run where selection was done once on the full table to see how much the leak inflated the number on your data.
- Online: a per-feature null-rate and distribution monitor on the serving path for every kept feature, alerting on a step change (Data & Feature Tests).
- Over time: on each retrain, diff the newly selected set against the deployed one; large churn in the set is itself a signal that the ranking is unstable and the features are near-interchangeable.
What can go wrong
- The nested selection is implemented once, correctly, and then someone "simplifies" the pipeline by selecting once up front on the full table because it is faster. The leak returns with a clean commit message.
- The feature set is frozen at selection time and never revisited; two years later half of the dropped features would now be strong and the kept ones have drifted, and nobody re-runs the selection because it is not in the retraining pipeline.
- A kept feature is removed upstream and the serving service fills it with a default. The model runs, the metric monitors see nothing until labels arrive a year later.
- Nested selection multiplies training cost by the number of folds and makes the pipeline harder to explain; the cheaper up-front selection is exactly the one that lies.
- Aggressive selection on serving grounds throws away features that carry real signal because their source is inconvenient; the honest response is to fix the source, which is another team's quarter.
- A stable, small feature set is easier to serve and monitor and makes the model more of a fixed hypothesis; it will not pick up a newly useful signal until someone re-opens the selection.
- "Selection improved the validation score, so the dropped features were noise." If the selection saw the validation rows, the score improved because the selection fitted them. Rerun it nested before believing the improvement.
- "The tree ensemble never split on it, so it is useless." It is useless to that ensemble given the other features present. Remove a correlated sibling and the ensemble may split on it constantly.
- "Fewer features means a weaker model." Usually a slightly weaker offline model and a considerably more reliable production one; the offline delta is rarely the number that decides.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a selection fitted on validation rows biases the validation metric follows from what a fitted procedure is; it applies to every selection method and every model family, and the size of the bias grows with the number of candidate features.
- MODEL-SPECIFICEmbedded selection is specific to the model that did it: an L1-penalised linear model zeroes features that are redundant *linearly*, a tree ensemble ignores features that are redundant *given the splits it chose*; neither verdict transfers to the other family, and a neural network on the same table does its own implicit selection that nobody can read.
- CONTESTEDA serious position holds that with a well-regularised tree ensemble explicit selection is unnecessary and slightly harmful: the ensemble already ignores what it does not need, selection removes features that would matter after drift, and the nested procedure adds cost for a metric gain that is usually inside the noise. That is right for offline quality; the argument for selection is that every unused feature is still a serving dependency and a skew surface, which the ensemble's indifference does nothing to remove.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a deploy-time check that every feature in the artifact resolves to a live source is a dependency test, and the practice of treating a silently defaulted feature as an incident rather than a warning is a reliability question this domain assumes rather than answers.