The Tuning Budget
A search costs compute, time and validation-set credibility, and returns less with every trial. Tune the learning rate first, stop when the curve flattens, and remember that a fixed leak or a better feature usually beats any amount of tuning.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How much should a hyperparameter search cost, what should it spend its first trials on, and when is the next trial worth less than the next feature?
A marketing team has been tuning a conversion model for six weeks. Each week's search improves the validation number by a little less than the week before. Meanwhile a data engineer has pointed out that one of the features is computed from the day after the conversion event, and nobody has had time to look.
Keep tuning while the number keeps going up. Each search is a fixed cost, the improvement is real, and stopping means leaving performance on the table.
The number is going up because the search is fitting the validation set a little better each week — three hundred trials times six weeks against one set. The held-out period, which nobody has checked since week one, has not moved (Metric Uncertainty).
- The number is going up because the search is fitting the validation set a little better each week — three hundred trials times six weeks against one set. The held-out period, which nobody has checked since week one, has not moved (Metric Uncertainty).
- Whatever real improvement the tuning found is dwarfed by the leaked feature. The model's validation score is inflated by information that will not exist at serving time, and tuning made the model better at exploiting the leak (Evaluation Leakage).
- Six weeks of cluster time have gone into a search whose largest single gain came in the first afternoon, when the learning rate was fixed. Every subsequent week bought less for the same price (Training Cost).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a visitor converts within seven days of a campaign touch. The label is a purchase event joined from the order system; the decision is which visitors get the expensive follow-up.
- The search objective is validation PR AUC, and the team reports its week-on-week improvement as progress.
- One example is one campaign touch with the visitor's prior sessions, referrer, device and a set of engagement aggregates. Three million rows.
- One of the engagement aggregates —
sessions_last_7d— is computed as of the export date, not the touch date, so for converted visitors it includes the sessions that led to the purchase (Temporal Leakage). - Each week's search is roughly three hundred trials of gradient boosting at twenty minutes each, on a shared cluster.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A search returns diminishing improvements because the objective is roughly unimodal in the settings that matter: the first trials move from a bad region to a good one, and the rest move within the good region, where the surface is flat. The gain per trial falls faster than the cost per trial does.
- The learning rate is first because it interacts with everything and has the widest effect: too high and nothing else matters, too low and every other setting is judged on an undertrained model. Fixing it early makes every later comparison meaningful. Capacity settings come next, and decision settings — the threshold — last, because they depend on the final model's score distribution (Threshold Selection).
- Tuning cannot fix a leak, a wrong label, or a missing feature, because those are properties of the data the search is optimising against. It can make a model exploit them more efficiently, which is why a leaked feature usually gets *more* important as the search proceeds.
Where the first trials go
The learning rate decides whether every other comparison is fair. At a rate too high, depth and regularisation are being compared on models that never converged; at a rate too low, on models that stopped early. Fix it first, with rounds chosen by early stopping on validation, and the rest of the search is a search over models that were actually trained.
Then capacity: depth, minimum leaf size, feature fraction, λ. Then, once and last, the threshold — which is chosen on the final model's score distribution and is meaningless before that model exists. Each stage gets a budget, and the record says what each stage bought.
The budget is forty trials. Which settings get them, and in what order?
when Always, and first. Rounds via early stopping, learning rate on a log scale over two or three orders of magnitude. Roughly a quarter of the budget.
cost A few trials that are obviously divergent; they are cheap and they bound the range for everything after.
when After the learning rate is fixed. Depth, leaf size, λ, feature and row sampling. The bulk of the budget.
cost Interactions with the learning rate are ignored; a small re-check of the learning rate at the end catches the worst case.
when Once, on the final model, on validation, from the cost of each error type. Not a search — a calculation.
cost One more use of the validation set; unavoidable, and the reason the test set exists.
when When trials are very cheap and the setting count is small; a joint random search can find interactions the staged one misses.
cost No stage-by-stage record; the winner is a point and the next model starts from scratch.
The number that always rises
Six weeks, three hundred trials a week, one validation set. The validation curve rises every week because the search keeps finding configurations whose errors line up a little better with that set's noise. The held-out curve, which no trial touched, stopped moving in week two. Nobody looked at it because it was not the number that changed.
The cost was not only cluster hours. It was six weeks in which the one change that would have mattered — removing a feature computed after the label — sat in a backlog, because the dashboard said progress was being made.
Validation PR AUC rose a little every week for six weeks. The team reported cumulative improvement as tuning progress.
The held-out period, checked in week six, is where it was in week two. The deployed model's follow-up precision is far below either number, because sessions_last_7d does not contain future sessions at serving time.
- 1The dominant effect is the leaked feature: the model's offline score is inflated by post-label information, and tuning made it lean on that feature harder.
- 2The weekly validation gains after week two are selection optimism from eighteen hundred trials against one set; the held-out period shows the real gain was small and early.
- 3A smaller contribution: the learning rate found in week one was re-tuned every week alongside everything else, so later weeks were partly re-finding the same region.
Better data beats a better search
A search optimises the model against the dataset as it is. It cannot add information the dataset lacks, and it cannot remove information the dataset should not have. A new aggregate that exposes a signal the model could not see before, or a fixed join that removes a leak, changes the objective; a search only moves within it.
This is the honest ranking of where an afternoon goes: a leak audit first, then the baseline, then a feature the domain expert has been suggesting, then the learning rate, then everything else. Tuning is real, and it is last, and it has a budget.
The validation metric the search is optimising measures what the model will be asked to do at serving time — no leaked features, labels that mean what the decision needs, a validation period that resembles deployment.
holds when A leakage audit has run on the feature set; every aggregate is computed as of the prediction time and not the export time; the held-out period agrees with the validation period on the baseline.
breaks when A feature is computed as of a later date than the prediction; a label is joined from a system that is updated after the fact; the validation period contains a campaign the deployment period will not.
respond Stop the search. Fix the data. Re-run the baseline. Only then decide whether the tuning record still applies, and expect that it does not.
How to build it
Most important first.
- Before the first search, audit the features for leakage and check the baseline (The Leakage Audit, Baselines Are Mandatory). A search on a leaked dataset is a precise optimisation of a fiction, and its cost is entirely wasted.
- Tune the learning rate first, with the number of rounds chosen by early stopping. Then capacity and regularisation. Then the threshold, once. Record the gain from each stage so the next model starts from what was learned (Experiment Tracking).
- Set the budget in advance, in trials or hours, and stop at it. Set a second, smaller budget for the second search and compare the gain per hour; when the second search bought less than a feature-engineering afternoon would, stop searching.
- Track the held-out period alongside the validation score every week. When validation rises and the held-out number does not, the search is overfitting the validation set and more budget will make it worse.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Gain on the untouched held-out period per hour of search, by week. This is the number that says whether the next week is worth running; the validation number cannot, because it always rises.
- The gain from each tuning stage — learning rate, capacity, threshold — so that next time the budget goes where it paid off.
- Do not measure progress as the week-on-week validation delta. That number is dominated by selection optimism once the trial count is in the hundreds.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The dataset the search optimised against is free of leakage and its labels mean what the decision needs; a search assumes its objective is honest, and cannot check.
- The validation set still resembles the deployment period after weeks of selection against it; the held-out period is the only evidence for this and it must be checked, not assumed.
- Trial cost is stable enough that a budget in hours means what it did when it was set; a cluster change that halves trial cost doubles the selection pressure at the same nominal budget.
- Offline: plot held-out score against cumulative search hours. A flat line with a rising validation curve is the signature of a search that has stopped finding real improvement.
- Online: the deployed model's conversion precision on the first weeks of real follow-ups, against the held-out estimate, not the validation one (Offline vs Online Evaluation).
- Over time: at each retraining, spend a small fixed budget re-checking the learning rate and rounds against the new data before any wider search; if the good region has not moved, the wider search is not needed.
What can go wrong
- The budget is set in trials, a faster machine is provisioned, and the trials get cheaper — so the team spends the saving on more trials against the same validation set rather than on stopping earlier.
- The learning rate was tuned at one number of rounds, then rounds was tuned separately, and the two now interact badly; the stages were treated as independent when they are not.
- The team fixes the leak, retrains with the tuned settings, and the number drops sharply. The drop is read as a regression and the leak is reinstated "until we can investigate".
- A fixed budget sometimes stops just before a real improvement. That cost is real, small, and much smaller than the cost of an unbounded search against one validation set.
- Auditing features before tuning delays the first search by days, and the audit finds nothing most of the time. When it finds something, it saves the entire search.
- Tuning in stages is slower than a joint search and can miss interactions; it is also legible, and the record it leaves is worth more than the interaction it misses.
- "The validation number is still going up, so we should keep tuning." It is going up because you are selecting against it. Check the held-out period; if it is flat, the search is finished whether or not the validation number knows it.
- "We can't fix the leak now, we'd lose all our tuning gains." The gains were made on a leaked objective. Fixing the leak reveals the model's real quality; the tuning is not lost, it was never real.
- "A bigger search would find a better configuration." Past the good region the surface is flat. A bigger search finds a configuration with luckier validation noise.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a search's gain per trial falls while its cost per trial does not applies to every model family and search method; where it flattens depends on the problem, and the held-out curve is how you find out.
- DATA-SPECIFICThe claim that a better feature beats tuning is strongest on tabular data with hand-built features, where a new aggregate can change what the model can see; on a pretrained network fine-tuned on images or audio the features are learned and the learning rate schedule really is the dominant lever.
- CONTESTEDSome practitioners hold that budgets should not be fixed in advance at all — that the right stopping rule is a statistical test on the held-out improvement, and that a fixed hour count is arbitrary and often stops a search that was about to pay off. That is a defensible position for teams with a rigorous held-out protocol; the fixed budget is a simpler rule for the far more common team that has no held-out protocol and a validation number that only ever rises.
Where the depth lives
This domain teaches the model and hands the rest off by name.