Trees & Ensembles

Decision trees, random forests and gradient boosting — how each learns, why boosting fits residuals, and XGBoost and LightGBM as implementations rather than as answers.

Decision Trees
▶ lab

A tree is a sequence of `feature < threshold?` questions ending in a constant. Readable, axis-aligned, piecewise flat — and incapable of predicting a value it has never seen.

Q · A decision tree is the most readable model there is. What does its shape let it express, what can it never express, and when does the readability stop being worth the limits?
How a Tree Chooses a Split
▶ lab

Gini, entropy or variance reduction score each candidate cut; the greedy search picks the best one at each node and recurses. Stopping rules are the only thing between that and a leaf per point.

Q · How does a tree decide which feature and which threshold to split on, why does growing it to completion memorise the data, and what does that imply for scaling and for categoricals?
Random Forests
▶ lab

Many deep trees, each on a bootstrap sample and a random feature subset, averaged. Variance falls because the trees disagree; the out-of-bag rows give a free validation estimate; the artifact is large.

Q · A single tree is high-variance. Why does averaging many of them help, why do the trees have to be different for it to work, and what does the ensemble cost at serving time?
Gradient Boosting

Fit a tree, compute the residuals, fit the next tree to them, repeat. Each tree follows the negative gradient of the loss; the learning rate shrinks each step; the number of trees is the capacity knob, chosen by early stopping.

Q · Boosting builds trees sequentially, each on the errors of the ensemble so far. Why is that a gradient descent, why does it overfit differently from a forest, and why does it find a leak before it finds anything else?
XGBoost and LightGBM as Implementations

Second-order gradients, a regularised objective, histogram binning, leaf-wise growth, native missing-value and categorical handling — what the fast implementations add to boosting, at the level of the mechanism and never the API.

Q · What do the production boosting libraries do that the textbook algorithm does not, and which of those choices changes how the model overfits, handles missing values or scales?
Tree Ensembles: When and When Not

A single tree, a forest, boosting and a linear model scored on quality, latency, cost, interpretability, data needed and operations — and the cases where boosting is the wrong answer even though it would win the benchmark.

Q · Boosting usually wins the offline tabular benchmark. When is it still the wrong model to ship, and what questions decide that before the benchmark is run?