Tree Ensembles: When and When Not
A single tree, a forest, boosting and a linear model scored on quality, latency, cost, interpretability, data needed and operations — and the cases where boosting is the wrong answer even though it would win the benchmark.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Boosting usually wins the offline tabular benchmark. When is it still the wrong model to ship, and what questions decide that before the benchmark is run?
Four teams in one company each ask "which model?" for a tabular problem: a pricing model that must extrapolate to prices never charged; a bidding model that scores ten thousand candidates in a few milliseconds; a clinical-risk model that must output probabilities a doctor can act on; and a new product with four hundred labelled rows. Each has been told the answer is the same library.
Gradient boosting wins on tabular data. Use the library, tune it, ship it. If it is not good enough, try a neural network.
The pricing model is asked about a price a fifth above anything in history. Every tree-based model returns the edge box — the demand at the highest price ever charged — and the elasticity beyond that is invisible. A linear model in log-price continues the trend; wrong, perhaps, but not silently flat (Decision Trees).
- The pricing model is asked about a price a fifth above anything in history. Every tree-based model returns the edge box — the demand at the highest price ever charged — and the elasticity beyond that is invisible. A linear model in log-price continues the trend; wrong, perhaps, but not silently flat (Decision Trees).
- The bidding model has two thousand boosted trees and a few milliseconds to score ten thousand candidates; the p99 blows the budget and the auction times out. A linear model scores the batch as one matrix multiply (Throughput vs Latency).
- The clinical model's boosted scores are sharp and uncalibrated; a doctor told "0.9" acts on it, and the reliability curve says the true rate at that score is much lower. The model needed calibration, and the calibration needed held-out data the team did not reserve (Calibration).
- The four-hundred-row model overfits the spreadsheet with any ensemble; the honest number is a regularised logistic regression's, and it is barely above the baseline — which is the real finding (Beating the Baseline).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Four different targets: a price elasticity, a bid value, a probability of an adverse event, and a churn flag on a tiny dataset. What they share is a tabular feature table and a team that wants one recommendation.
- What differs is the decision each feeds — extrapolation, a latency-bound ranking, a calibrated probability, and a sanity check — and it is the decision, not the data format, that chooses the model.
- All four are tabular with engineered features, which is the regime where tree ensembles are strong. Data sizes range from hundreds of rows to hundreds of millions; latency budgets from batch to sub-millisecond per item; interpretability requirements from none to regulatory.
- None of the four has clean labels; the clinical model's labels come from coded records with known omissions, the tiny dataset's from a spreadsheet.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The four families differ on axes the benchmark does not score. Extrapolation: trees and everything built from them predict a constant beyond the training range; linear models continue a trend. Latency: a linear model is one dot product; a forest or a boosted ensemble is hundreds or thousands of dependent tree walks per item. Calibration: logistic regression outputs are probabilities by construction; boosted scores are monotone in risk but need a post-hoc map to be probabilities. Data needed: a linear model with a handful of features fits stably on hundreds of rows; an ensemble needs thousands to control variance.
- Quality on a large, clean, tabular benchmark is where boosting wins, and the margin is real. The question is whether the deployment cares about the axes where it loses, and the answer is a set of questions — what does the decision consume, what is the latency budget, must the model explain itself, does the input range move, how many rows are there — asked before any model is trained (Which Model Should We Use?).
- Interpretability is not binary. A single shallow tree is a flowchart; a linear model is a set of coefficients; a forest or boosted ensemble is explainable only through post-hoc methods with their own caveats. Which is "enough" depends on who has to act on the explanation and whether they can challenge it.
Six axes, four families
The offline benchmark scores one column. The deployment is scored on six. A model that dominates on quality and fails on latency does not ship; one that dominates on quality and cannot be explained to the regulator does not ship; one that dominates on quality and outputs uncalibrated scores to a doctor ships and does harm.
The matrix below is the typical shape for tabular data with engineered features. Every cell is an ordinal judgement to be replaced with a measurement — but the shape is what the reflex answer ignores.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Single decision tree | A flowchart; cheap to serve; unstable across retrains and rarely the strongest model. | ||||||
| Random forest | Stable, forgiving defaults, large artifact; latency scales with the number of trees. | ||||||
| Gradient boosting | Usually the strongest on clean tabular data; leak-seeking; needs early stopping and calibration; many knobs. | ||||||
| Regularised linear model | One dot product; calibrated by construction; extrapolates; needs the representation done by hand. |
caveat The scores are for the typical tabular case and say nothing about extrapolation, calibration or a specific latency budget — each of which can move a family from best to unusable. "Quality" here is the offline benchmark; a family that wins it can lose on production outcomes for every reason the rest of this domain describes. Replace every cell with a measurement from your own held-out data and load test before deciding anything.
Four cases where boosting loses
The pricing model needs to answer a question outside its data. Trees cannot; they return the edge box. The bidding model needs ten thousand scores in a few milliseconds; a thousand trees per candidate does not fit, a dot product does. The clinical model needs probabilities a person acts on; boosted scores are not probabilities without a calibration step and the data to fit it. The four-hundred-row model cannot support an ensemble's variance at all.
None of these is a failure of the algorithm. Each is a constraint the benchmark does not score, discovered by asking what the decision consumes before asking which model scores highest. In each case the answer was known before training anything.
Which constraints exclude which families?
when Pricing, capacity planning, any input the business intends to push past history.
cost Tree ensembles are out; a linear or parametric model continues the trend and must be range-monitored because it will continue it into the unknown.
when Real-time bidding, candidate scoring at retrieval time, embedded scoring.
cost A large ensemble is out or must be distilled; the quality margin lost should be quantified, not assumed away.
when Clinical risk, credit, any output read as a probability by a human or a policy.
cost A logistic model is calibrated by design; an ensemble needs a calibration step, its own held-out data, and a reliability monitor.
when A few hundred rows, a new product, a rare event.
cost Ensembles overfit; a regularised linear model or a rule is honest, and its number may show the model should wait for data.
The choice has a shelf life
A family chosen under constraints is correct until the constraints move. The latency budget tightens with a product redesign; the price range extends with a strategy change; a regulator asks for explanations that were not required last year; the tiny dataset grows into a large one. The model may be unchanged and the choice wrong.
So the recommendation is not a model but a record: which questions were asked, what the answers were, and which families they excluded. When an answer changes, the record says what to revisit — and prevents the reflex answer from being rediscovered as if it were new.
The latency budget, input range, interpretability requirement, data volume and consumed metric that excluded some families and selected this one are still the constraints the deployment operates under.
holds when The product, the regulatory context and the serving infrastructure are stable, and the record of the choice is reviewed at each material change.
breaks when A latency budget tightens, a price band extends past history, an explanation becomes mandatory, or the dataset grows past the point where the small-data choice still applies.
respond Re-run the family comparison under the new constraints with the linear baseline included; a change of family is a new model with its own promotion path, not a retrain (Promotion Is a Checklist, Not a Score).
How to build it
Most important first.
- Ask the five questions first — decision, latency, interpretability, data size, metric — and let them exclude families before any benchmark runs (The Questions Before the Boxes).
- Run the linear baseline regardless. It is the reference the ensemble must beat *by enough to pay for its costs*, and on tiny or extrapolating problems it is the answer (The Linear Baseline).
- When boosting is the right family and probabilities are consumed, reserve calibration data and measure the reliability curve as a promotion criterion, not an afterthought.
- When latency excludes the ensemble, consider distilling it into something servable and keeping the ensemble as the offline reference — the comparison is then quality lost against budget met (Pruning & Distillation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The metric the decision consumes, per candidate family, alongside p99 latency at production batch size and the calibration error if probabilities are used. One number per family is not a comparison.
- The margin over the linear baseline against the ensemble's operational cost — retraining time, artifact size, serving fleet — in units the business recognises.
- Do not measure "which model is best" on a single offline metric. That is the question the brief forbids, and it produces the same answer every time regardless of the problem.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The constraints that chose the family — latency budget, input range, interpretability requirement, data volume — remain as they were; a product change that tightens latency or extends the price range can invalidate the choice without touching the model.
- The calibration map, where one exists, still maps scores to observed rates; it is fitted to a score distribution that drifts.
- The margin over the baseline that justified the ensemble's cost persists on production outcomes, not just on the offline benchmark.
- Offline: the matrix, measured — each candidate family's decision metric, p99 latency at production batch size, reliability curve and artifact size on the same held-out data.
- Online: the margin over the baseline on production outcomes, and the latency and calibration numbers under real load; a family that wins offline and loses on any hard constraint is not the winner.
- Over time: re-ask the five questions at each significant product change; the answer to "which model" has a shelf life set by the constraints, not by the data.
What can go wrong
- The trade-off matrix is filled in from folklore rather than measured; the ensemble's latency is assumed fine, and the surprise arrives in the load test after promotion.
- The linear model is chosen for extrapolation and extrapolates confidently into a regime where the trend does not hold; no model can know that, and the fix is a range monitor plus a policy, not a model.
- Calibration is applied once and never re-checked; the score distribution drifts with the data and the "probabilities" quietly stop being probabilities (Prediction Drift).
- Asking the questions first costs time before any model is trained and sometimes yields the unglamorous answer — a regularised linear model — that is hard to present as progress.
- Excluding a family on a constraint forgoes its quality; a linear model chosen for latency leaves the ensemble's margin on the table, and that margin has a business value that should be stated alongside the budget it violated.
- Distillation recovers part of the margin under the budget at the price of two models to maintain and a quality gap that must be re-measured at every retrain.
- "Which model? XGBoost." Without the questions about data, latency, interpretability, metric and cost, that is a reflex, not a recommendation; it is right often enough to be dangerous.
- "If boosting is not good enough, a neural network will be." On tabular data with engineered features, a tuned ensemble is usually the strongest single model and a network is rarely better; "neural networks are always better" is false in exactly this regime.
- "The ensemble's scores are probabilities." They are monotone in risk. A reliability curve is the test, and boosted models typically fail it until calibrated on held-out data.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat model choice depends on the decision, latency, interpretability, data and cost — not on an offline metric alone — holds for every task and family; the specific rankings in the matrix are what changes.
- DATA-SPECIFICThe matrix is scored for tabular data with engineered features, where tree ensembles are strong; on images, audio or free text the quality column inverts and a pretrained network's learned representation wins by a margin no tuning of trees recovers.
- CONTESTEDA respectable position holds that on tabular data one should simply default to a tuned boosted ensemble, because it wins the quality benchmark so consistently that the exceptions — extrapolation, tiny data, strict latency — are rare enough to handle when they arise, and asking five questions before every model is process for its own sake. That is a reasonable default for a team with many models and one platform; it is wrong whenever one of the exceptions is the actual problem, which is more often than the benchmark literature suggests.
- SIMPLIFIEDThe matrix scores are ordinal judgements for typical tabular deployments, not measurements; a real comparison replaces each cell with a number from the team's own held-out data and load test.
Where the depth lives
This domain teaches the model and hands the rest off by name.