Overfitting
A model that memorises the noise in its training set scores perfectly on that set and poorly on the next one. Every route to a better training number is also a route to this.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The training metric keeps improving as you add capacity, epochs and features, and the validation metric stopped improving a while ago. What has the model learned since then, and why does a leaked feature look like the opposite?
A support team wants to route incoming tickets to the right specialist queue. The data scientist's model classifies the historical tickets almost perfectly. In its first week live it routes a third of tickets wrongly, and the misroutes are not random: it is confident about tickets that resemble specific old ones.
Add capacity until the training accuracy is as high as it can go. A model that can classify every historical ticket has clearly captured what distinguishes the queues.
Past a certain point the model stops learning what distinguishes queues and starts learning what distinguishes *tickets* — a customer id, a phrase from one incident, a typo in a template. Those features perfectly separate the training set and mean nothing about the next ticket.
- Past a certain point the model stops learning what distinguishes queues and starts learning what distinguishes *tickets* — a customer id, a phrase from one incident, a typo in a template. Those features perfectly separate the training set and mean nothing about the next ticket.
- With a random split, near-duplicate tickets land on both sides, so the validation set rewards memorisation too: the memorised phrase is in validation as well. Offline accuracy is high; the model has learned to recognise old tickets, not classify new ones (Entity Leakage).
- In production, every ticket is new. The memorised features are absent or coincidental, and the model's confident predictions rest on nothing. The failure is only visible once the resolved-queue labels arrive, days later.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict which of six specialist queues a new ticket belongs in. The label is the queue where the ticket was finally resolved, which is one to five days after it was opened.
- The decision is a routing action taken at ticket creation; a wrong route costs a hand-off and a day of delay, so the number that matters is accuracy on tickets that do not exist yet.
- One example is one ticket: subject, body, customer tier, product, and a bag of engineered text features. Forty thousand tickets over eighteen months, with the resolved queue joined on as the label.
- Tickets are not independent. The same customer files near-duplicate tickets; agents paste template replies into bodies; a product incident produces hundreds of nearly identical tickets in one afternoon.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model with enough free parameters can fit any labelling of its training inputs, including a random one. Fitting the training set is therefore not evidence of learning a general rule; it is evidence that the model is flexible enough. What separates rule from memory is whether the fit transfers to rows that were not part of it.
- Three knobs raise training fit and each is a route to overfitting. Capacity (depth, width, degree) lets the model represent more functions, including ones that go through every noise point. Epochs let an optimiser keep reducing training loss after the generalising structure has been found, by fitting the residual noise. Features each add a dimension in which some training points can be separated by accident; with more features than rows, perfect separation of noise is guaranteed.
- The train/validation gap is the observable. It opens when the model begins to fit training-specific structure, which is invisible from the training side because from that side it is just more fit.
What the model learns after it stops learning
Watch training and validation loss per epoch. Early on both fall: the model is finding the structure that separates queues. Then validation loss flattens while training loss keeps falling. Everything learned after that point separates training tickets from each other without separating queues — a customer's phrasing, an incident's template, a typo.
The same picture appears along the capacity axis and the feature axis. Add tree depth and the leaves shrink toward one ticket each; add features and eventually some accidental combination isolates every training row. Fit rises, transfer falls, and the training side cannot see the difference.
1def fit_and_gap(train_fn, eval_fn, X_tr, y_tr, X_val, y_val, settings):2 # Sweep one flexibility knob and record BOTH errors at each setting.3 rows = []4 for s in settings:5 model = train_fn(X_tr, y_tr, s)6 tr = eval_fn(model, X_tr, y_tr)7 va = eval_fn(model, X_val, y_val)8 rows.append((s, tr, va, va - tr))9 # choose on validation error, and refuse settings where the gap10 # is still widening even if validation happens to tick down11 best = min(rows, key=lambda r: r[2])12 return best, rowsThe fourth column is the point. A setting where validation error is lowest but the gap is large is a model that is barely holding on; a neighbour with slightly worse validation and a much smaller gap will usually degrade more gracefully when the data shifts.
epoch train_loss val_loss what the model is fitting 1 1.62 1.60 nothing yet 5 0.71 0.78 queue-level vocabulary 12 0.38 0.52 product / tier interactions 20 0.19 0.51 <-- validation stops improving here 40 0.06 0.58 customer phrasing, template quirks 80 0.01 0.71 individual tickets
Why leakage looks like the opposite of overfitting
Overfitting has a fingerprint: training strong, validation weak. Leakage has the inverse: training strong, validation strong, production weak. The two are confused because both end in a bad product, but they are diagnosed by different numbers and fixed by different actions.
In the ticket data, assigned_agent is populated when a ticket is routed — after the prediction would be made. It is a near-perfect proxy for the queue. A model with it available scores brilliantly on both splits and has learned nothing about ticket text. Regularising it would be pointless; the feature has to go (Target Leakage).
looks like A categorical column present on every historical ticket, strongly predictive, and the first thing an importance ranking puts at the top.
why it leaks Agents belong to queues. The column is written when a human routes the ticket — the very decision the model is supposed to make — so it encodes the label with a small delay.
fix Rebuild features as of ticket creation time and drop anything written by the routing process itself. Then expect the honest model to show a train/validation gap — that is what a real fit on noisy text looks like.
The number that was measuring the wrong thing
The routing model was evaluated on a random split of tickets. Near-duplicate tickets from the same incident sat on both sides, so the validation set was partly a test of recognition rather than classification. The number was correct about that test and silent about the production task.
The honest evaluation groups by incident and customer. It gives a lower number that the business finds disappointing, and it predicts production behaviour. Which of those two facts matters is not a modelling question.
Validation accuracy 0.93 under a random ticket-level split; training accuracy near 1.0. Read as a small, acceptable gap.
Roughly a third of tickets misrouted in week one; the misroutes concentrate on tickets with no close match in the training set.
- 1Near-duplicate tickets from the same incidents and customers were split across training and validation, so validation rewarded memorised phrasing exactly as training did.
- 2The model had enough capacity to isolate individual tickets, and the epoch count was chosen where training loss stopped falling rather than where validation loss did.
- 3A smaller part of the gap is genuine novelty: two new products launched between the training cutoff and go-live and their tickets resemble nothing in training.
Every feature the model uses generalises across tickets: none of them identifies a specific training example or a specific incident so precisely that the model can memorise through it.
holds when High-cardinality identifiers are excluded or hashed coarsely, incident-level duplicates are grouped in the split, and the honest validation number is close to the random-split one.
breaks when A new feature with per-customer or per-incident cardinality enters the pipeline, or a burst of duplicate tickets from one incident dominates the next training set.
respond Find the feature or the duplicate cluster; do not regularise harder to compensate. Regularisation fights capacity, not identifiers.
How to build it
Most important first.
- Split so that memorisation cannot be rewarded: group by customer or by incident, so the validation set contains no near-copies of training rows (Group Split).
- Pick capacity, epoch count and feature set on validation, never on training. Track the gap explicitly and treat a widening gap as a signal in its own right (Learning Curves).
- Constrain the fit: regularisation, early stopping, dropout, tree depth limits, minimum samples per leaf — each is a way of refusing part of the training fit (Regularisation, Early Stopping).
- Add data before adding capacity. More rows make the noise harder to memorise and the rule easier to find; more capacity does the reverse.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Validation error under a split that respects the data's grouping, and the gap between it and training error. That pair decides how much fit to allow.
- Production accuracy once resolved-queue labels arrive, sliced by whether the ticket resembled a training ticket. A model that is right on look-alikes and wrong on novel tickets has memorised (Evaluation Slices).
- Do not measure training accuracy as a quality number at all. Its only use is as one side of the gap.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- New tickets are drawn from the same population as the training tickets, so the structure that generalised in validation still generalises — an incident-shaped spike of novel tickets breaks this on the day it happens.
- The features the model relied on are general, not identifying: no feature functions as a lookup of a specific training row (a customer id, a ticket number, a hash of the body).
- The validation set remains free of near-copies of training rows at every retrain, so the gap it reports continues to mean what it meant.
- Offline: compare validation accuracy under a random split against a grouped split. A large drop between them is memorisation being unmasked, and the grouped number is the honest one.
- Online: for the first week, compute accuracy on tickets with no near-duplicate in training separately from the rest. If the model only works on look-alikes, it did not learn the task.
- Over time: plot the train/validation gap at each retrain. A gap that grows as the model is tuned toward better training numbers is the drift toward overfitting made visible.
What can go wrong
- The group split is done but the groups are wrong — customers are separated while incidents are not — and the validation set still contains hundreds of near-duplicates of training tickets.
- Regularisation is tuned on validation until the gap closes, then the same validation set is used to report the final number, which is now optimistic by the amount of the search (Evaluation Leakage).
- The team reads a great validation number as "not overfitting" when it is a leak. A feature that encodes the answer gives strong training *and* strong validation, which is the one pattern overfitting never produces.
- Grouped splits shrink the effective validation set and make the reported number noisier and lower. It is lower because it is honest.
- Every constraint on the fit — regularisation, early stopping, depth limits — gives up some real structure along with the noise. The bias/variance trade-off is not avoided by choosing the variance side.
- Adding data is the cleanest fix and the one least under the engineer's control; it is often a labelling budget question rather than a modelling one.
- "Validation accuracy is high, so it is not overfitting." Under a random split of grouped data, validation rewards the same memorisation training does. The split decides what the number can detect.
- "Add a few more features — they can only help." Each feature is another dimension in which noise can be separated. Past the point where the model can already fit the noise, features hurt generalisation and help nothing.
- "Great validation, great training — we are done." That pattern is also the fingerprint of a leaked feature. Overfitting produces a gap; leakage produces none. Audit the features before promoting.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a sufficiently flexible model fits noise and that the fit does not transfer is a property of fitting, not of any model family; trees, networks and high-degree polynomials all do it.
- DATA-SPECIFICOn i.i.d. data a random split does detect overfitting; the failure described here needs grouped or duplicated rows — customers, incidents, sessions — which most business data has and most benchmark data does not.
- SIMULATEDThe routing accuracies in the offline/online device are chosen to show the shape of the gap between a random-split number and production; they are not measurements on any real ticket system.
Where the depth lives
This domain teaches the model and hands the rest off by name.