The Linear Baseline
Logistic or linear regression as the first real model: cheap to fit, cheap to serve, readable, and the reference for everything after it. If the complex model cannot beat it clearly, the complexity is not earning anything.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a regularised linear model on the same features score, and what does its distance from the complex model tell us about where the signal is?
An insurance team wants to predict which new claims will need an adjuster visit. They have tabular claim data and a proposed gradient-boosted model. The compliance officer wants to know why each claim was flagged, the platform team wants it to run inside a CPU service with a tight latency budget, and the lead wants to know whether the boosted model is doing anything a simpler one would not.
Go straight to the boosted model; it is the strongest tabular learner and will beat a linear model anyway. If interpretability is required, add an attribution method afterwards. If latency is tight, optimise the serving later.
The regularised logistic regression lands close to the boosted model on the decision metric. The gap is inside the interval. The boosted model's extra capacity found little the features do not express additively.
- The regularised logistic regression lands close to the boosted model on the decision metric. The gap is inside the interval. The boosted model's extra capacity found little the features do not express additively.
- The compliance officer is shown attribution charts for the boosted model and asks whether they are exact; they are not. The logistic model's coefficients are exact, and a per-claim contribution is coefficient times feature, which the officer can audit (Explainability).
- The boosted model fits the latency budget only after tuning tree count and depth; the logistic model is a dot product and fits with room to spare, in any language the platform team prefers (Latency Breakdown).
- Nobody built the linear model until asked, so all of this was discovered after the boosted model was proposed for promotion, with the attribution tooling already bought.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a claim will require an in-person adjuster visit. The label is the visit decision, recorded when the claim is assigned.
- The decision is routing: visit queue or desk review. A wrong routing costs a delay either way; an inexplicable routing costs a compliance finding.
- One example is one claim at filing: claim type, amount, policy tenure, prior claims, property features, and free-text length as a crude proxy.
- A few hundred thousand claims over three years; features are mostly categorical and numeric with modest interactions.
- The boosted model and the linear model are trained on the same encoded feature matrix with the same time-based split.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A linear model scores each example as a weighted sum of its features plus a bias; logistic regression passes that sum through a sigmoid to get a probability (Logistic Regression). Every feature's contribution is its coefficient times its value, so a prediction decomposes exactly into per-feature terms.
- What it cannot express is an interaction — that claim amount matters differently for different claim types — unless the interaction is hand-built as a feature. A tree ensemble learns interactions from data; the linear model's distance from it is therefore a measure of how much interaction structure the data contains (Gradient Boosting).
- Regularisation is what makes it a fair baseline: an L2 penalty on scaled features keeps coefficients bounded and stable on correlated columns; without it, and on unscaled features, the model is both worse and less stable and loses to everything (Regularized Linear Models).
- At serving, a linear model is a dot product: microseconds on a CPU, trivially portable, no runtime dependency on a training framework. Its serving cost is the cost of fetching the features.
A weighted sum, and what it cannot express
Logistic regression scores a claim as a bias plus the sum of coefficient times feature, then squashes the sum to a probability. Each claim's score is exactly the sum of its per-feature terms, so "why was this claim flagged" has an exact answer: these terms were large. No approximation, no baseline choice, no attribution method.
The price is that the model cannot learn that claim amount means something different for a water claim than for a theft claim, unless someone builds the amount × type feature. That is the whole gap between it and a tree ensemble, and the gap's size on the holdout is a measurement of how much interaction structure the data has.
1import numpy as np2 3# 1. scale on the training fold only; the scaler ships with the model4mu, sd = X_train.mean(axis=0), X_train.std(axis=0) + 1e-95Z_train, Z_val = (X_train - mu) / sd, (X_val - mu) / sd6 7# 2. logistic regression with an L2 penalty, fitted by gradient descent8def fit_logistic(Z, y, l2, steps=2000, lr=0.1):9 w, b = np.zeros(Z.shape[1]), 0.010 for _ in range(steps):11 p = 1 / (1 + np.exp(-(Z @ w + b)))12 grad_w = Z.T @ (p - y) / len(y) + l2 * w13 grad_b = (p - y).mean()14 w, b = w - lr * grad_w, b - lr * grad_b15 return w, b16 17# 3. tune the penalty on the validation fold like any hyperparameter18best = max(19 ((l2, score(fit_logistic(Z_train, y_train, l2), Z_val, y_val)) for l2 in [1e-4, 1e-3, 1e-2, 1e-1]),20 key=lambda t: t[1],21)22# 4. per-claim explanation is exact: contribution_j = w_j * z_jSteps 1 and 3 are what make it a fair baseline. Skip the scaling and the coefficients fight the units; skip the tuning and the default penalty is whatever the library chose for someone else's data.
Reading the gaps in the table
With the constant, the rule, the linear model and the candidate in one table on one holdout, the gaps are the findings. Constant to linear is what the features carry additively. Linear to candidate is what interactions and non-linearity add. If the second gap is inside the interval, the candidate has not demonstrated a reason to exist beyond the linear model.
The gaps are also a guide to effort. A large constant-to-linear gap and a small linear-to-candidate gap says the value is in the features, and the next unit of work is a better feature, not a bigger model. The reverse says the interactions are where the signal is and feature work will not recover them.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Regularised logistic regression | Within noise of the candidate on this holdout; a dot product to serve; exact per-claim contributions; one scaler to ship. | ||||||
| Gradient-boosted ensemble | Slightly stronger point estimate; fits the latency budget after tuning; explanations are approximate; a framework runtime in the serving path. | ||||||
| Logistic regression + three hand-built interactions | Closes most of the quality gap on this data while staying exact to explain; the interactions are a maintenance commitment. |
caveat The quality scores are one holdout on one dataset and their difference is inside the interval; on data with strong interactions the first row drops to 2 and the comparison inverts. Interpretability here means exactness of decomposition, not that the coefficients are causal.
The reference model for every later comparison
Whichever model is promoted, the linear baseline stays in the retraining pipeline as the reference. Its number on each new holdout anchors the candidate's: if both fall, the month was harder; if only the candidate falls, the candidate regressed. And if the gap between them closes over time, the serving system the candidate needed has stopped paying for itself.
If the linear model is what ships — and on this data it plausibly is — the same discipline applies in reverse: the boosted candidate is re-fitted at each retrain so that a reopening gap is noticed, and the serving path carries the scaler as part of the artifact.
The relationships between features and the label remain close enough to additive that the linear model's performance stays within the margin that justified choosing it, or that justified not choosing the candidate.
holds when The claim mix and the feature definitions are stable; the linear-to-candidate gap is re-measured at every retrain and stays inside the interval.
breaks when A new claim type with different amount dynamics enters the book; an upstream feature is redefined; interaction structure appears that the additive model cannot express.
respond Add the specific interaction as a feature if it is nameable; otherwise revisit the candidate with the serving and interpretability costs re-stated, not assumed away.
Unscaled numerics, default encoding, no regularisation, no tuning, fitted in five minutes to have a row in the table. Loses to the boosted model by a wide margin, which is reported as evidence for the boosted model.
Scaled on the training fold, same encoding as the candidate, L2 penalty tuned on the validation fold, known interactions added as features, scaler shipped in the artifact. Lands within noise of the boosted model and is promoted on latency and exactness.
A baseline is only a reference if it is the best simple model that could be built; a deliberately weak one does not measure the candidate, it flatters it.
How to build it
Most important first.
- Build the linear baseline on the same encoded features and split as the candidate; scale numerics on the training fold and ship the scaler in the artifact (Preprocessing Lives in the Artifact).
- Regularise, and tune the strength on the validation fold like any hyperparameter; report the tuned version, not the default.
- If a few interactions are known to matter, add them as explicit features; a linear model with the right three interactions is still a linear model and still exact to explain.
- Put its row in the table next to the constant, the rule and the candidate, and read the gaps: constant-to-linear is what the features contain; linear-to-candidate is what interactions add.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The decision metric at the operating threshold for linear and candidate, on the same holdout, with intervals; the gap is the case for the candidate's complexity.
- Serving latency and cost per prediction for both, and whether each meets the budget without tuning.
- Do not measure the boosted model against an unregularised, unscaled logistic regression with default encoding. That is a weak baseline built to lose.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The feature relationships stay close enough to additive that the linear model's small distance from the candidate persists; if new interaction structure appears in the data, the gap reopens.
- The scaler and encoding in the artifact are applied identically at serving time, so the coefficients meet the feature values they were fitted on.
- The interpretability the linear model was chosen for is preserved by whoever maintains it: no unexplained interaction features are added later without the compliance reading being updated.
- Offline: linear and candidate in the same table on the same holdout with resampled intervals; the linear coefficients reviewed for sign and magnitude sanity by a domain expert before promotion.
- Online: a serving contract test that the scaler and encoding produce the same vector as training on a probe set (Serving Contract Tests); latency at the tail measured against the budget.
- Over time: on every retrain, the candidate is re-fitted too, so the gap is tracked; a widening gap is the signal to revisit which model to serve.
What can go wrong
- The linear model needs the scaler at serving time and the serving path skips it; the coefficients are applied to raw values and every prediction is off (Train / Serve Skew).
- The linear model is promoted for interpretability and the compliance team reads the coefficients as effects of the features on claims, rather than as the model's weights (Attribution Is Not Causality).
- The gap between linear and boosted was small on this year's data; the data changes, interactions appear, and the linear model — now the champion — is not re-compared against a candidate because the comparison was a one-off.
- A linear model caps at additive structure; when interactions genuinely matter it loses by a margin that no amount of regularisation recovers, and hand-building interactions scales badly.
- Interpretability is exact and shallow: a coefficient tells you the weight, not why the weight is what it is, and the compliance reading of it needs the same causal caution as any importance chart.
- Building and tuning the linear model properly is a day that produces something the team may not deploy.
- "A linear model is too simple for real problems." On tabular data with mostly additive structure it is frequently within noise of the best model; the way to find out is to build it properly, not to assume.
- "We tried logistic regression and it was much worse." Scaled? Regularised and tuned? Same encoding as the boosted model? If not, the comparison was against a model built to lose.
- "The boosted model is only slightly better, but slightly better is better." Not once the serving cost, the attribution tooling and the compliance risk are on the other side of "slightly".
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICOn tabular data with mostly additive relationships a tuned linear model is often within noise of a tree ensemble; on images, text and audio, or on tabular data with strong interactions, the linear model on raw features loses decisively and its value is in showing that, not in being a candidate.
- SCALE-SPECIFICUnder a tight CPU latency budget or at very high prediction volume the linear model's dot-product serving cost is a decisive advantage; with a relaxed budget and low volume the serving argument disappears and the comparison is on quality and interpretability alone.
- CONTESTEDA serious position holds that on tabular data a well-regularised gradient-boosted model is the right default and the linear baseline is a detour: boosting handles interactions, missing values and mixed types without feature work, and modern attribution methods give per-prediction explanations that satisfy most reviewers. That is a strong position for teams with the tooling and the latency headroom; the reply is that the linear row costs a day, and when it lands within noise it removes a serving system and a class of explanation caveats entirely.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a domain expert reviewing coefficient signs before promotion is a semantic review of a model the way a code review is of code; deciding what makes such a review binding is a process question this domain assumes rather than answers.