Decision Trees
A tree is a sequence of feature < threshold? questions ending in a constant. Readable, axis-aligned, piecewise flat — and incapable of predicting a value it has never seen.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A decision tree is the most readable model there is. What does its shape let it express, what can it never express, and when does the readability stop being worth the limits?
A lending team needs a model whose every decision they can explain to a regulator and to the applicant. They want the reasoning as a flowchart. Their first tree looks perfect on paper; in production it approves an income bracket it never saw during training exactly as if it were the highest bracket it did see.
Grow a decision tree. It is interpretable, needs no scaling, handles mixed feature types, and the flowchart is exactly what the compliance team asked for.
The tree's prediction is the constant at whichever leaf the applicant lands in. An applicant with revenue far above anything in training falls through the revenue >= 2.1M branch into the same leaf as the largest training loans and gets the same probability — the tree cannot extrapolate, only assign to a region.
- The tree's prediction is the constant at whichever leaf the applicant lands in. An applicant with revenue far above anything in training falls through the
revenue >= 2.1Mbranch into the same leaf as the largest training loans and gets the same probability — the tree cannot extrapolate, only assign to a region. - A fully grown tree has a leaf for nearly every training loan and its "reasoning" is a memorised path. It scores brilliantly on training and poorly on new applicants, and the flowchart the regulator sees is forty levels deep (Overfitting).
- Retrain on a slightly different quarter and the top split changes from revenue to years-trading; the flowchart the team explained last month is now a different flowchart. The explanation was of a fit, not of the world.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a small-business loan will default within twelve months. The label is a missed-payment event, known only a year after origination.
- The decision is approve, decline or refer; a declined applicant may ask why, so the model's output must be explainable in terms of the applicant's own features.
- One example is one originated loan with the applicant's features at application time — revenue, years trading, sector, requested amount, existing debt — and the default outcome a year later.
- A few tens of thousands of loans. Revenue and amount are continuous and skewed; sector is categorical with a long tail; the default rate is low.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A tree partitions feature space with axis-aligned cuts. Each internal node holds one feature and one threshold; a row goes left if
feature < thresholdand right otherwise. Each leaf holds a constant — the majority class or the mean label of the training rows that reached it. Prediction is a walk from the root to a leaf, comparing one feature per level. - Because every cut is on one feature, the regions are rectangles (boxes in higher dimensions) and the prediction is flat inside each one. A diagonal boundary needs a staircase of many cuts; a smooth relationship becomes a step function; and outside the range of the training data the nearest box extends forever, so the prediction is whatever that box held.
- Growth is greedy: at each node, choose the feature and threshold that best reduce impurity for the rows at that node, then recurse into the children. Nothing looks ahead, so a split that is weak alone but powerful in combination is missed, and depth is the only way to express interactions (How a Tree Chooses a Split).
A sequence of one-feature questions
The whole model is the diagram: a root question, two children, more questions, then leaves holding a constant. To score an applicant you start at the top and follow the answers. Each question looks at exactly one feature and compares it to one number; the path taken is the explanation, and the leaf is the prediction.
That structure is why the data structure and the model are the same thing. The tree is a Binary Tree whose internal nodes are (feature, threshold) pairs and whose leaves are values; prediction is a root-to-leaf walk with a depth-bounded number of comparisons, which is why a single tree is one of the cheapest models to serve.
revenue < 480k?
├── yes: years_trading < 2?
│ ├── yes: existing_debt_ratio < 0.35?
│ │ ├── yes: [default rate 0.11]
│ │ └── no: [default rate 0.34]
│ └── no: [default rate 0.06]
└── no: requested_amount < 900k?
├── yes: [default rate 0.03]
└── no: [default rate 0.05] <-- every applicant above 900k lands here, at any revenueBoxes, staircases and the edge that never ends
Every leaf is a rectangle in feature space and every prediction inside it is the same number. A relationship that is smooth — default risk falling gently with revenue — becomes a staircase, and the number of steps is bounded by depth. A boundary that runs diagonally, such as "risk depends on debt relative to revenue", needs many axis-aligned cuts to approximate, which is why a ratio feature helps a tree more than it helps a linear model.
The last box on every axis has no far wall. An applicant with ten times the largest training revenue is inside the same box as the largest training loan, and the tree returns that box's constant with the same confidence it would for a typical member. That is the no-extrapolation property, and it is a property of every tree-based model — forests and boosting inherit it exactly.
Risk falls with revenue; the tree encodes this as five steps and then a flat line from the last cut to infinity. A revenue of 50M scores exactly as 2.1M did.
One coefficient expresses the trend and continues it outside the training range — for better or worse. A revenue of 50M produces a risk lower than anything observed.
The tree's output is bounded by the leaf values it has seen; the linear model's is not. Which is safer depends on whether the world continues the trend, which the data cannot say. The tree is honest about its ignorance in one way — it never invents a value — and dishonest in another: it never signals that it is guessing.
The explanation is a sample
A single tree is a high-variance estimator: small changes in the training set move the greedy split choices, and a different top split reshuffles everything beneath it. The flowchart is therefore one draw from a distribution of flowcharts, and presenting it as *the* reasoning overstates what was learned.
That does not make the tree useless as an explanation. It makes stability something to check and report: the splits that persist across refits are the ones the data supports; the ones that come and go are the fit talking. A shallow, stable tree fitted to the predictions of a stronger model is a common and honest compromise.
Applicants at prediction time fall within the ranges of the split features seen in training, so the leaf they land in describes loans like theirs rather than the nearest edge.
holds when The applicant population is stable, feature definitions are unchanged, and the training window covered the full range of the segments the product serves.
breaks when A new product tier brings applicants with revenue or amounts beyond the training range; a reporting change rescales a feature; a marketing push reaches a sector with a handful of training rows.
respond Refer out-of-range applicants to human review rather than trusting the edge box; then extend the training window or the product's data before retraining.
How to build it
Most important first.
- Limit depth and minimum leaf size on validation, not by taste. A shallow tree is honest about what it knows; a deep one is a lookup table (Regularisation).
- Treat the tree's explanation as a description of *this fit* and check its stability across resamples before presenting it as the reasoning (Explainability).
- Know where the training data ends and guard against inputs outside it, because the tree will confidently extend the edge box. Range checks at serving time are a model-invariant test, not paranoia (Model Invariant Tests).
- If a single tree cannot reach the required quality at an explainable depth, move to an ensemble for prediction and keep a shallow tree as the explanation surface — two different jobs (Random Forests).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Validation quality against depth, and the depth at which it peaks. The visualiser at
/ml/treedraws training and held-out accuracy by depth; the divergence is the whole lesson. - Split stability: how often the same top-level features appear across bootstrap refits. If the flowchart changes every retrain, it is not an explanation.
- Do not measure interpretability by the fact that a tree *can* be printed. A forty-level tree is printable and unreadable.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Applicants at prediction time fall inside the feature ranges the tree was grown on; outside them, its prediction is the nearest edge box and carries no information about the new region.
- The thresholds still separate what they separated: a change in how revenue is reported (gross versus net) moves every applicant across cuts that were tuned to the old definition.
- The categorical values seen at serving time were seen in training; a new sector value has no branch and takes whichever default path the implementation chooses.
- Offline: the depth sweep on a held-out set; and a stability check of the top splits across bootstrap refits before any flowchart is shown to a regulator.
- Online: monitor the fraction of requests that fall outside the training range on any split feature; those predictions are extrapolations and should be flagged or referred.
- Over time: compare the tree structure across retrains. Structural churn with stable quality is variance in the explanation, and it should change how the explanation is presented.
What can go wrong
- The tree extrapolates flat: a new applicant segment with out-of-range revenue is scored as the edge box, and no monitor sees it because the input is valid and the output is in range (Robustness Testing).
- Depth limited to what fits on one slide, and the tree now underfits — it cannot express the interaction between sector and years-trading that decides most defaults (Underfitting).
- Sector, with hundreds of values, is one-hot encoded and the tree spends its depth on single-sector splits that isolate a handful of loans each; the rare sectors are memorised.
- Readability is bought with axis-aligned boxes and flat predictions, so a tree spends depth on things a linear model expresses in one coefficient — a smooth trend costs a staircase.
- No extrapolation is a safety property as much as a limit: the tree never invents a wild value, but it also never notices it is out of its depth.
- A single tree is high-variance by construction; the fix for that — an ensemble — costs the readability that motivated the tree.
- "The tree is interpretable, so we understand the decision." The tree is a fit. Refit it and the top split may move; what is being interpreted is one sample of a high-variance estimator.
- "Trees handle any input." They handle any input by mapping it to a training-time box. An input beyond the training range gets a confident, uninformed answer.
- "A deeper tree is a smarter tree." A deeper tree has smaller leaves, and a leaf with one loan in it is a memory, not a rule. Depth is chosen on validation.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAxis-aligned partitioning with a constant per leaf is what a CART-style tree is, whatever the impurity measure or the library; the inability to extrapolate follows from the constant leaves and holds for every tree ensemble built from them.
- SIMULATEDThe Decision Tree Visualizer grows a real CART tree on seeded synthetic 2D points; its training-versus-held-out accuracy by depth shows the mechanism and is not a measurement on any lending data.
- DOMAIN-SPECIFICIn regulated lending, an explainable decision is a legal requirement and a shallow tree is sometimes the right model at a real quality cost; in ad ranking or fraud, the same tree is only ever a baseline or an explanation aid for a stronger ensemble.
Where the depth lives
This domain teaches the model and hands the rest off by name.