Random Forests
Many deep trees, each on a bootstrap sample and a random feature subset, averaged. Variance falls because the trees disagree; the out-of-bag rows give a free validation estimate; the artifact is large.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A single tree is high-variance. Why does averaging many of them help, why do the trees have to be different for it to work, and what does the ensemble cost at serving time?
A property portal's valuation model is a single decision tree whose estimates swing wildly between monthly retrains — the same flat is valued a fifth higher or lower depending on which month's data trained it. Agents have stopped trusting it. The team wants estimates that are stable and does not want to hand-engineer features.
One tree overfits, so grow many trees on the same data and average them. More trees, more stable.
Trees grown deterministically on the same data are identical. Averaging a hundred copies of one tree is that tree; the variance is untouched.
- Trees grown deterministically on the same data are identical. Averaging a hundred copies of one tree is that tree; the variance is untouched.
- Trees grown on bootstrap samples but with the full feature set are *nearly* identical: the strongest feature (location) wins the top split in every tree, and everything beneath is similar. Averaging highly correlated estimators reduces variance only a little.
- The forest fixes the instability and the team ships it — into a serving path with a tight latency budget where two hundred deep trees per request is a different cost from one (Latency Breakdown).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the sale price of a listed property from its attributes and location. The label is the achieved price, known at completion, weeks to months after listing.
- The decision is a suggested asking price; a wildly unstable suggestion is worse than a slightly biased one because agents override anything that looks arbitrary.
- One example is one completed sale: size, rooms, age, energy rating, latitude/longitude, distance to transport, and neighbourhood aggregates. A few hundred thousand sales over five years.
- Location effects are strongly nonlinear and interacting — the same square footage is worth very different amounts a street apart — which is why the team wants a model that finds interactions on its own.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Averaging
nestimators with equal varianceσ²and pairwise correlationρgives varianceρσ² + (1 − ρ)σ²/n. Asngrows the second term vanishes and the first remains. So averaging only helps to the extent the trees are *decorrelated*; the forest's design is entirely about loweringρ. - Two sources of randomness do that. Bootstrap sampling: each tree trains on a sample drawn with replacement, so about a third of rows are absent from any given tree and the trees see different data. Feature subsampling: at each split, only a random subset of features is considered, so the dominant feature cannot win every top split and the trees are forced to find different structure.
- Each tree is grown deep — individually overfit, low-bias, high-variance — because averaging handles the variance and bias would not be averaged away. Prediction is the mean (regression) or the vote (classification) across trees. The rows left out of each tree's bootstrap are its out-of-bag set, and predicting each row only from trees that did not see it gives a validation estimate without a held-out split (Cross-Validation).
Averaging only helps if they disagree
The arithmetic is the argument. Average n estimators that each have variance σ² and correlate with each other at ρ, and the variance of the average is ρσ² + (1 − ρ)σ²/n. The second term goes to zero as trees are added. The first does not move. A hundred identical trees have ρ = 1 and the average is one tree; a hundred independent trees would have ρ = 0 and variance σ²/100.
Real forests sit between, and the whole design — bootstrap rows, subsample features at every split — exists to push ρ down. Feature subsampling is the surprising part: deliberately hiding the best feature from most splits makes each tree worse and the average better, because the trees stop agreeing about the top of the tree.
1import numpy as np2 3def random_forest(grow_tree, X, y, n_trees=200, max_features=None, seed=0):4 rng = np.random.default_rng(seed)5 n, p = X.shape6 k = max_features or max(1, int(np.sqrt(p)))7 trees, oob_sum, oob_cnt = [], np.zeros(n), np.zeros(n)8 for _ in range(n_trees):9 idx = rng.integers(0, n, size=n) # bootstrap: with replacement10 # grow_tree receives a feature-subset sampler it must call at EVERY split11 tree = grow_tree(X[idx], y[idx], choose=lambda: rng.choice(p, k, replace=False))12 trees.append(tree)13 oob = np.setdiff1d(np.arange(n), idx) # rows this tree never saw14 oob_sum[oob] += tree.predict(X[oob]); oob_cnt[oob] += 115 oob_pred = oob_sum / np.maximum(oob_cnt, 1)16 return trees, oob_pred # compare oob_pred to y for the estimateThe feature subset is drawn per split, not per tree. Drawing once per tree still lets the dominant feature win the root of every tree that received it; drawing per split is what breaks the correlation at the top.
The free estimate and what it cannot see
Each bootstrap leaves out roughly a third of the rows. Predict each row using only the trees that did not train on it and you have a held-out estimate computed from the training data alone. For independent rows it tracks a proper validation split closely, and it costs nothing.
It is a row-level random split, and it fails the same way. If the same building appears in fifty sales, a tree that saw forty-nine of them predicts the fiftieth well for reasons that will not hold for a building it has never seen. OOB is a first look; a grouped held-out set is still the number.
Out-of-bag error comfortably below the single tree's, and stable across retrains — the improvement the team wanted.
On genuinely new developments — buildings with no prior sales — production error is substantially higher than OOB suggested, and estimates for those cluster at the edge of the training price range.
- 1OOB rows were near-copies of in-bag rows from the same buildings; the estimate was partly measuring recognition of the building, not valuation of the property.
- 2New developments sit outside the trained boxes on location and price; every tree extrapolates flat and the average of flat extrapolations is a flat extrapolation.
- 3A smaller part is genuine drift: the market moved in the months between training cutoff and the completions being scored.
What the ensemble costs to serve
A single tree is one root-to-leaf walk. A forest of two hundred deep trees is two hundred walks, each a chain of data-dependent branches through a structure far larger than any cache line. The compute per prediction is small; the memory traffic and the unpredictable branches are what set the latency, and they scale with the number of trees.
That is a serving question, not a modelling one, and it is the assumption most likely to break when a notebook forest meets a production budget. The number of trees is a knob with a quality plateau and a linear latency cost; choose it with both curves in view.
The deployed forest fits in serving memory and returns within the per-request latency budget at production concurrency, at the number of trees chosen for quality.
holds when Inference runs in batch, or the online budget is generous relative to trees × depth, or a distilled model stands in for the forest on the hot path.
breaks when Trees are added for a marginal quality gain, the serving fleet is downsized, or a new feature deepens every tree; latency grows linearly and the tail grows faster.
respond Cut trees back to the quality plateau, cap depth, or distil; do not accept the tail-latency regression as the price of stability without measuring the alternative.
How to build it
Most important first.
- Use the defaults as a starting point — deep trees,
√pfeatures per split for classification,p/3for regression, a few hundred trees — and confirm on validation; the forest is unusually forgiving of them (Hyperparameters). - Read the out-of-bag error as a first check, then verify it against a proper held-out split that respects grouping: OOB is row-level and is fooled by grouped or duplicated rows exactly as a random split is (Group Split).
- Budget the artifact and the inference before promotion: trees × depth × features is memory, and per-request latency scales with the number of trees (Inference Cost).
- Where the latency budget cannot afford the forest, distil it — a shallower tree or a boosted model fitted to the forest's predictions — and keep the forest as the offline reference (Pruning & Distillation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Held-out error under a grouped split, and its spread across retrains — the stability the team asked for is the standard deviation of the prediction for a fixed property across monthly models.
- Out-of-bag error as a cheap internal estimate; it should track the grouped held-out error, and a gap between them is a sign the rows are not independent.
- p99 inference latency and artifact size at the chosen number of trees. Do not measure quality alone when the number of trees is the knob (Tail Latency: Why p50 Being Fine Does Not Help).
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The rows remain roughly independent at the level the bootstrap samples them; if the data becomes dominated by repeated listings of the same units, the trees are less decorrelated than the design assumes.
- The serving infrastructure continues to afford the artifact: memory for the trees and latency for walking all of them per request.
- The training distribution of prices and locations still covers the properties being valued; the forest does not extrapolate beyond it.
- Offline: OOB error against grouped held-out error; the number-of-trees curve to confirm error has plateaued; and prediction stability across bootstrap refits for a fixed panel of properties.
- Online: p99 latency under production load with the full forest, and the out-of-range counter on location and size features.
- Over time: the month-to-month prediction change on the fixed panel — the number the agents were complaining about — as a monitored metric.
What can go wrong
- The forest inherits every tree's inability to extrapolate; a new luxury development beyond the training price range is valued at the edge box, averaged over two hundred edge boxes (Decision Trees).
- OOB error is quoted as validation on data where the same building appears many times; the OOB rows are near-copies of in-bag rows and the estimate is optimistic (Entity Leakage).
- Impurity-based feature importance from the forest is presented to agents as "what drives price"; it is biased toward high-cardinality and continuous features and says nothing about causation (Attribution Is Not Causality).
- Stability is bought with size: a forest is hundreds of deep trees, and the artifact can be gigabytes where the single tree was kilobytes.
- Every prediction walks every tree, so latency grows linearly with the ensemble and the tight-budget serving path may need a distilled substitute.
- Readability is gone. The forest cannot be drawn, and its importance measures are biased; the explanation surface has to be built separately.
- "More trees always helps." More trees reduce the
σ²/nterm toward zero; they do nothing aboutρσ². Past a few hundred, error is flat and latency keeps rising. - "OOB error means we do not need a validation set." OOB is a row-level random split. On grouped data it has the same leak as any random split, and a grouped held-out set is still required.
- "The forest is robust to everything." It is robust to the individual tree's variance. It is not robust to leakage, to distribution shift, to out-of-range inputs or to a serving budget it does not fit.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALVariance reduction by averaging decorrelated estimators is a statistical fact independent of the base learner; bagging works for any high-variance model, and the forest is the case where the base learner is a deep tree.
- DATA-SPECIFICOn tabular data of moderate size a forest with defaults is a strong, stable baseline; on images or text without engineered features it has nothing to split on and a learned representation wins decisively.
- SCALE-SPECIFICAt batch-inference scale the forest's size is a storage question and its latency is irrelevant; at online scale with a tight per-request budget the same forest may be unservable without distillation or a smaller ensemble.
Where the depth lives
This domain teaches the model and hands the rest off by name.