Don't Delegate Understanding
Libraries hide optimisation, AutoML hides search, feature stores hide synchronisation, model servers hide inference, cloud platforms hide infrastructure, foundation models hide training. Use all of them — and know what each one is hiding when it breaks.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Every layer of ML tooling hides something on purpose. Which things must the engineer still understand about a model they did not write, and what does "Model Accuracy = 94%" need before it means anything?
A team fine-tuned a foundation model with a managed platform, served it through the platform's endpoint, and reported a headline accuracy to leadership. Leadership approved a launch. The engineer who ran the pipeline cannot say what data the base model was trained on, which split produced the number, or what the platform does when a feature is missing.
The platform is built by experts and handles the hard parts. The engineer's job is to supply data and read the metric. Understanding the internals is what the platform exists to make unnecessary.
The random split put tickets from the same customer thread in both training and evaluation, so the number measures memorisation of threads. Production tickets are new threads and the routing quality is far lower (Entity Leakage).
- The random split put tickets from the same customer thread in both training and evaluation, so the number measures memorisation of threads. Production tickets are new threads and the routing quality is far lower (Entity Leakage).
- The platform silently imputes a missing
product_areafield with the most common value; in production the field is missing for every ticket from the new mobile app, and they all route to the desktop queue. - The base model was trained on public text that ends before the company's newest product existed; tickets about it are routed by surface similarity to an old product. No amount of fine-tuning data about the new product was present to correct it.
- When the number turned out to be wrong, nobody could say which of these it was, because every layer that would have shown it was hidden by design.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model classifies support tickets into routing queues; the label is the queue a human eventually routed the ticket to. The headline number was the platform's default evaluation.
- The target of the lesson is a list of questions that must have an answer whoever built the model, and the habit of asking them before believing a number.
- The fine-tuning set is a few months of tickets with their final queue. Tickets that were re-routed twice carry only the last queue; tickets closed without routing are absent.
- The base model's training data is undisclosed. The platform's evaluation split its held-out set at random from the same months.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Each layer of tooling replaces a mechanism with a default. A library replaces the optimiser with a default learning rate and stopping rule; AutoML replaces model selection with a search over a space someone chose; a feature store replaces the synchronisation of training and serving features with a promise; a model server replaces the inference path with a container; a cloud platform replaces the cluster with a bill; a foundation model replaces an enormous training system with weights and a card.
- The defaults are usually right, which is why the tools are worth using. They are wrong in ways that are specific to the data and the decision, and the tool cannot know either. The engineer who does not know what the default replaced cannot tell when it is wrong.
- What must be understood is not the internals but the assumptions: what data the model learned from, what objective it optimised, what it depends on staying true, and what happens when that stops. Those four questions are answerable for a black-box model; they are just not answered by the box (Learning vs Programming).
What each layer hides
The table is not an argument against the tools. Each replaces a mechanism most teams would implement worse. It is a list of what to know about each one *anyway*, because the mechanism it hides has assumptions and the tool does not know your data.
The last column is the habit: for each layer, the one question whose answer you must be able to give about a model you did not write.
| Layer | Hides | Assumes about your data | Must still know |
|---|---|---|---|
| ML library | Optimisation: learning rate, stopping rule, regularisation defaults | Scale, size and noise level of a generic dataset | Which objective was optimised, and whether it stopped early or late |
| AutoML | Search: which models, which hyperparameters, which metric | The default metric is your metric; the search space contains the right model | What space was searched, on which split, for which metric |
| Feature store | Synchronisation of training and serving features | The definition registered is the one both paths compute; timing agrees | Whether point-in-time correctness is enforced or promised |
| Model server | Inference: batching, preprocessing, fallbacks, versions | The preprocessing in the container matches the one at training | What the server does with a missing or unknown input |
| Cloud ML platform | Infrastructure: clusters, accelerators, storage, orchestration | The default split, imputation and evaluation suit the problem | Which defaults fired, and how the reported number was produced |
| Foundation model | An enormous training system and its data | The pretraining data covers your domain and time period | The training cut-off and coverage, and which of your slices it cannot have seen |
What "94%" needs before it means anything
The number leadership approved was Model Accuracy = 94%. The domain's habit is to hear that and not nod. Every question below has an answer that could turn the number into a different one, and in the ticket-routing case three of them did.
Ask them in this order, because each narrows what the number can mean. By the time the last is answered, the number is either a claim about the decision or it has been replaced by one that is.
- On what dataset — which months, which tickets, and which tickets are absent because they were never routed?
- How was it split — at random, by time, by customer thread? A random split on threaded data measures memorisation.
- What is the class balance — if most tickets go to one queue, what does always predicting that queue score?
- Is there leakage — does any input carry information from after the routing decision?
- Against which baseline — the current keyword rules, a majority-class predictor, the previous model?
- Which metric matters for the decision — accuracy across queues, or the cost of mis-routing the queues that are expensive to get wrong?
- At what threshold, and how calibrated — will the routing act on every prediction or only confident ones, and does the confidence mean anything?
- In production — how are the features produced there, and does the imputation match?
- Is it reproducible — can the team get the number again, outside the platform, on a split it chose?
- What happens when the data changes — a new product, a new client with missing fields, a reorganised queue?
1# What a metric has to carry to be a claim rather than a number.2report = {3 "metric": "accuracy", # and why this one maps to the routing decision4 "value": 0.94, # illustrative5 "dataset": "tickets 2026-03..2026-06, routed only", # absent: never-routed tickets6 "split": "random 80/20", # <- the problem: threads straddle the split7 "class_balance": {"billing": 0.61, "technical": 0.27, "other": 0.12},8 "majority_baseline": 0.61, # what always-billing scores9 "previous_system": "keyword rules", # and its number on the same split10 "threshold": None, # routing acts on argmax; no abstain path11 "features_at_serving": "platform default imputation for product_area",12 "reproduced_outside_platform": False,13}Written out, the number answers itself: a random split over threaded tickets, a majority baseline that already scores most of the way there, no reproduction, and a hidden imputation at serving. None of that was visible in "94%".
The layer that hides the most
The foundation model is the largest delegation in the stack: someone else's data, someone else's objective, someone else's cut-off. Fine-tuning adjusts the surface; it does not add knowledge of a product the pretraining data never mentioned. The assumption that the base model covers your domain is the one most teams never write down, and it is the one that failed on the new-product tickets.
It is also the assumption that the tool most actively obscures, because a model card describes the data in general terms and a fine-tuning service reports only the fine-tuning metric. The detection has to be built by the team: a slice of the evaluation made of exactly the things the base model could not have seen.
The pretraining data of the foundation model contains enough about the concepts in your inputs — products, terms, entities — that fine-tuning can align its representation with your labels.
holds when The domain is public and older than the model's training cut-off, and the fine-tuning evaluation includes a slice restricted to the newest concepts and scores acceptably on it.
breaks when A product, term or entity appears after the cut-off or was never public; the model routes it by surface similarity to something older, confidently.
respond Add labelled examples for the uncovered concepts and re-evaluate on the slice; if the base model still cannot separate them, the problem is representation, not fine-tuning, and a retrieval step or a different base model is the fix.
How to build it
Most important first.
- For every tool in the pipeline, write down what it hides and what its default assumes about your data — the split, the imputation, the metric, the stopping rule — and check each against the decision (AutoML Hides the Search, Feature Stores).
- Never accept a metric without its provenance: dataset, split strategy, class balance, baseline, metric choice, threshold, calibration. A number missing any of them is a number without a claim (Metric Uncertainty, Business Metrics vs Model Metrics).
- For a foundation model, read what is known about the training data and its cut-off, and test the slices of your problem it could not have seen (Foundation Models, Transfer Learning).
- Reproduce the number once yourself, outside the platform, on a split you chose. If it does not reproduce, the platform's number is the platform's, not yours (Reproducibility).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Quality on a split that mirrors production — new customer threads, the new product, missing fields — at the threshold the routing decision uses. This is the number that maps to the decision; the platform's default is the number that maps to the platform's default.
- The fraction of production inputs that hit each hidden default: imputed fields, out-of-vocabulary values, fallbacks. A default that fires on a large share of traffic is a decision the engineer did not make.
- Do not measure understanding by whether the pipeline can be re-run. It can always be re-run. The question is whether its number can be explained.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every hidden default in the pipeline — imputation, encoding, split, stopping rule, fallback — still assumes something true about the production data, and a change to the tool's version has not changed the default.
- The base model's training cut-off and coverage still include the concepts the tickets are about, or the fine-tuning data covers what it does not.
- The platform's evaluation split still reflects how production tickets differ from training tickets, and the reproduced number still holds on a split the team chose.
- Offline: rebuild the evaluation on a group-aware, time-aware split outside the platform and compare with the platform's number; the gap is the size of what the default hid.
- Online: log which hidden defaults fired per request — imputed fields, fallbacks — and alert when the rate moves; a new client sending a missing field is a default becoming the model.
- Over time: on each platform or base-model version change, re-run the reproduced evaluation before the change is allowed to reach production.
What can go wrong
- The team writes down what each tool hides, and the tool changes its default in a version bump the platform applies automatically (The Model Supply Chain).
- The engineer reproduces the number outside the platform, gets a lower one, and the platform's number is the one that has already been presented.
- Understanding is delegated to one person who knows the base model's limits, and they leave; the artifact stays and the assumptions go with them.
- Understanding what each tool hides is the time the tool was meant to save; a team that must document every default before shipping is slower than one that trusts them, and is right to be only when the defaults are wrong.
- Reproducing the platform's number outside the platform means maintaining a second evaluation path, which drifts from the first.
- Questioning a headline number after it has been presented is politically expensive, which is why the questions must be asked before.
- "Understanding the model means reading the training code or the architecture paper." It means knowing what data it learned from, what objective it optimised, what it assumes and what happens when the assumption fails. That is answerable for a black box.
- "The platform's evaluation is unbiased because the platform has no stake." The platform's split, metric and imputation are defaults chosen for a generic problem; they are not biased, they are uninformed about the decision.
- "Foundation models make the training data question moot." They make it someone else's answer, which the engineer still has to ask; a model that has never seen the new product will route its tickets wrongly however good the fine-tuning.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a tool replaces a mechanism with a default that assumes something about the data is true of every layer from a scikit-learn imputer to a managed fine-tuning service; only the size of what is hidden changes.
- SIMPLIFIEDThe headline "94%" in this lesson is invented to illustrate the shape of the questions; no real platform, dataset or split is being measured, and the numbers exist only so the questions have something to attach to.
- CONTESTEDA serious position holds that this is nostalgia: modern platforms encode more expertise than a typical team has, their defaults beat most hand-rolled choices, and demanding the team understand every layer produces slower, worse systems. That is right about the defaults on average; the counter-argument is that the failures are not average, they are specific to the data and the decision, and the platform cannot know either.
Where the depth lives
This domain teaches the model and hands the rest off by name.