Feature Importance
Split gain, coefficients and every other model-specific importance answer one question: what does this model use? They do not answer what matters in the world, and unscaled coefficients do not even answer the first one.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model reports an importance for every feature. What does that number actually measure, why do two models on the same data disagree about it, and what is it safe to conclude from it?
A retention team's churn model has a "feature importance" chart on the dashboard. The product lead sees that support_tickets_90d is at the top and plan_price is near the bottom, and concludes that pricing does not affect churn. A second model trained on the same data by another engineer ranks them the other way round.
Read the importance chart the library produces. The feature at the top is the one that matters most; a feature near zero can be ignored, or dropped, or reported to the business as irrelevant. If two models disagree, the more accurate one is right about importance too.
The logistic regression's coefficient on plan_price is tiny because a one-dollar change is a tiny change; the coefficient on usage_ratio is large because a one-unit change is the whole range. The chart is ranking units, not influence.
- The logistic regression's coefficient on
plan_priceis tiny because a one-dollar change is a tiny change; the coefficient onusage_ratiois large because a one-unit change is the whole range. The chart is ranking units, not influence. - The tree ensemble puts the 90-day usage aggregate at the top and the 30-day one near zero — because they carry the same signal and the ensemble split on whichever it met first. Dropping the "unimportant" one on that evidence changes nothing; dropping the "important" one changes nothing either, because its sibling takes over.
- Split-gain importance in the ensemble favours high-cardinality and continuous features, which offer more split points. A near-random continuous column can out-rank a binary feature that carries real signal.
- The product lead announces that pricing does not affect churn. Price has barely varied in the training data — everyone is on one of two plans — so no model can measure its effect; the chart reported an absence of variation as an absence of importance.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a subscriber will cancel within the next 60 days. The label is the cancellation event, observed at the end of the window.
- The decision is which accounts a retention specialist calls this week, so what is needed is a ranking of accounts, not a ranking of features. The feature chart is a by-product that the business has started to read as a finding.
- One example is one subscriber at the start of a month: plan price, tenure, usage aggregates over 30 and 90 days, support tickets, and whether a discount is active.
- Two models were trained on the same table: a gradient-boosted ensemble, and a logistic regression on the raw, unscaled columns.
plan_priceis in dollars and ranges over a few hundred;usage_ratiois between zero and one. - Several features are near-duplicates of each other — three usage aggregates over overlapping windows — so the signal in "usage" can be carried by any of them.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A coefficient in a linear model is the change in the log-odds per unit of the feature. Its magnitude depends on the unit; standardise the features and the coefficients become comparable, otherwise the chart ranks by scale (Bucketing & Normalisation).
- Split-gain importance in a tree ensemble sums, over every split on the feature, the reduction in loss that split achieved on the training data (How a Tree Chooses a Split). It measures how much the training procedure used the feature — on training data, with the other features present, subject to which of several correlated features happened to be chosen.
- Both are statements about the fitted model. They say how the model's output depends on its inputs. A feature that does not vary in the data has no importance under any method, whatever it does in the world; a feature that proxies for the label has the highest importance, and that is the leak talking.
Coefficients rank units unless you scale them
A logistic regression coefficient says how much the log-odds move per unit of the feature. plan_price in dollars has a small coefficient because one dollar is a small step; usage_ratio in [0, 1] has a large one because one unit is the whole range. Ranking the raw coefficients ranks the units the engineers happened to store the columns in.
Standardising each feature to unit variance makes coefficients comparable — each is now "log-odds per standard deviation" — and moves the scaler into the artifact, because the same transform must run at serving time. The interval matters as much as the point: a coefficient whose interval crosses zero has not been measured, and "not measured" is not "small".
1# raw columns: coefficients are per-unit, and the units differ2w_raw = fit_logistic(X_train, y_train)3# plan_price 0.004 (per dollar; range is a few hundred dollars)4# usage_ratio 2.100 (per unit; the whole range is one unit)5 6# standardised on the training fold only, scaler shipped in the artifact7mu, sd = X_train.mean(axis=0), X_train.std(axis=0)8w_std = fit_logistic((X_train - mu) / sd, y_train)9# plan_price 0.30 (per standard deviation of price)10# usage_ratio 0.25 (per standard deviation of usage)11 12# the ranking flipped; the model is the same function of the dataThe two fits are the same decision boundary expressed in different coordinates. Only the second chart is comparable across features, and only with an interval next to each bar.
Split gain measures use on the training set
A tree ensemble's importance sums the loss reduction from every split on a feature, on the data the tree was grown on. Three things follow. It is a training-set number, so a feature the trees used to memorise noise scores well. It prefers features with many candidate split points, so continuous and high-cardinality columns are favoured. And among correlated features it credits whichever was split on first, so the credit for "usage" lands on one sibling and the others read as useless.
None of this makes the number wrong; it makes it an answer to a narrow question. "How much did the training procedure lean on this column, with these siblings present" is the question. "What matters" is not.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Three overlapping usage aggregates | One ranks first, two rank near zero | The ensemble split on one and the others became redundant given it; the credit is arbitrary among siblings. | Report the group; test dropping a feature by refitting without it, not by reading the bar. |
Continuous account_age_days vs binary has_discount | Age out-ranks the discount flag | Split gain favours features with many split points; the binary feature can be used once per path. | Compare with permutation importance on held-out data, which does not reward split-point count. |
| A feature that barely varies in training data | Importance near zero; business concludes it is irrelevant | No variation, no measurable effect under any method; the chart reports the dataset, not the world. | State the range the feature took in training; outside it the model has no opinion. |
| An unexpectedly dominant feature | One bar dwarfs the rest | Most often a leaked proxy for the label rather than a discovery. | Leakage audit before the chart leaves the notebook. |
What the chart is allowed to say
An importance chart is a description of the fitted model. Read that way it is useful: it tells you which features the serving path must get right, which are load-bearing for the current weights, and which changes upstream would hurt. The retention team can be told that the model leans heavily on support-ticket counts, and that the ticket pipeline is therefore a dependency to protect.
The chart cannot say that tickets cause churn, that pricing does not, or that closing the ticket queue would keep anyone. That inference needs an intervention or a causal design with stated assumptions. The next two lessons take the two halves: Permutation Importance for a more honest version of "what does the model use", and Attribution Is Not Causality for why even the honest version stops short of "what moves the outcome".
The importance chart on the dashboard describes the model currently serving, on features scaled and correlated as they were when the chart was made.
holds when The chart is regenerated at every retrain from the promoted artifact; the scaler in the artifact is the one used to compute coefficient importance; the correlated siblings the model leans on are all still arriving.
breaks when The model is retrained and the chart is not; a sibling feature breaks and the model silently shifts weight to another; the dashboard shows a chart made from a notebook model that was never the one promoted.
respond Investigate the feature that moved before anyone reads the new ranking as a finding; a reorder with no code change is a data change.
A bar chart titled "Churn drivers" on the executive dashboard, with support tickets at the top and price at the bottom, read as a finding about customers.
The same bars, titled "Features the current churn model depends on (v14, permutation importance, held-out, grouped by correlation)", with a note that price barely varied in training and its effect is unmeasured.
The first label licenses a business decision the chart cannot support. The second describes exactly what was computed and lets the reader ask for an experiment when they want the causal question answered.
How to build it
Most important first.
- Standardise before reading coefficients, and report them with intervals. A coefficient whose interval spans zero is not "small"; it is unmeasured.
- Prefer Permutation Importance on held-out data to split gain when the question is "what does the deployed model depend on", and read correlated features as a group rather than one at a time.
- Label the chart as what it is: "what this model uses", never "what drives churn". The Attribution Is Not Causality distinction has to be on the dashboard, not in a footnote.
- When the business question is genuinely "would changing X change churn", stop reading importance and design an intervention — an experiment, or a causal analysis with stated assumptions (Causality vs Prediction).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The stability of the ranking across bootstrap resamples and across model families. A ranking that reorders under resampling is noise; report the group of features that are consistently used, not their order.
- For a decision like "drop this feature", the held-out metric with and without it — the only number that answers that question directly.
- Do not measure "which features matter to the business" from any importance chart. It is not a measurement of that.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The importance was computed on data with the same feature scaling and encoding the serving path uses; a change in scaling changes coefficient-based importance without changing the model's behaviour.
- The correlation structure among features is stable, so the feature the model leans on is still present and still carries the shared signal; if a sibling breaks, the "important" feature's meaning changes.
- The audience reading the chart knows it describes the model and not the world, and the chart's labelling continues to say so after the engineer who made it has moved on.
- Offline: compute the ranking on twenty bootstrap resamples and across two model families; report only what is stable, and check the top feature against a leakage audit before anyone presents it.
- Online: log the ranking at every retrain and diff it; a reorder at the top with no code change means a feature or a source changed underneath.
- Over time: for any feature the business acts on because of the chart, look for whether the action changed the outcome; if the retention team stopped calling high-ticket accounts and churn among them did not move, the chart was read as causal and was not.
What can go wrong
- The chart drives feature deprecation: the "unimportant" 30-day aggregate is removed from the pipeline; the next retrain finds the 90-day one has drifted, and the model has lost the sibling that would have covered for it.
- A leaked feature ranks first, is celebrated as a discovery, and becomes the reason the project is funded. The leak is found at rollout.
- The importance chart is stable and the model is retrained monthly; nobody notices when the ranking changes, because no monitor compares it to last month's. A feature quietly took over from a broken sibling (Feature Drift).
- Reporting groups of correlated features instead of a ranked list is honest and disappointing; the business wanted a top three.
- Standardising features for interpretable coefficients adds a fitted step that must ship in the artifact and match at serving time (Preprocessing Lives in the Artifact).
- Bootstrapping the ranking costs as many training runs as resamples; the alternative is a single ranking whose stability is unknown.
- "Feature importance proves causality." It proves the model used the feature. A feature can rank first because it is a consequence of churn, a proxy for a leaked label, or the one of three siblings the tree met first.
- "The coefficient on price is tiny, so price does not matter." The coefficient is per dollar. Per standard deviation of price it may be the largest in the model — or unmeasurable, because price barely varied.
- "The more accurate model is right about importance." Two models of similar accuracy can use entirely different features to reach it, because the features are redundant. Accuracy does not select among equally good explanations.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- MODEL-SPECIFICCoefficient importance exists only for linear models and depends on scaling; split-gain importance exists only for trees and depends on which correlated sibling was split on first and on training-data loss; a neural network offers neither, and gradient-based attributions on it have a different set of caveats again.
- DATA-SPECIFICOn a table with many correlated aggregates the ranking is nearly arbitrary among siblings; on a table of a few independent, well-scaled features the ranking is stable and close to what permutation importance reports, and the caveats in this lesson mostly stop mattering.
- CONTESTEDA serious position holds that model-specific importance is still the right default: it is free, it is exactly what the fitted model does, and permutation methods have their own artefacts on correlated features (they create impossible input combinations). That is a fair criticism of permutation importance; the reply is that split gain is measured on training data and rewards leaked and high-cardinality features, so it should be one input to a stability check rather than the chart on the dashboard.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — diffing the importance ranking across retrains is a regression test on a model property rather than on a metric, and deciding what size of reorder should block promotion is a test-design question this domain assumes rather than answers.