Decision Before Model
The model exists to change a decision. Name the decision, its owner, its capacity and its moment first, and most model choices — target, features, metric, threshold, inference mode — are made for you.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why does naming the decision before the model fix so many later choices, and what happens to a model built for a decision nobody named?
A logistics company: "We want to predict which shipments will be damaged. We have inspection reports going back years." Asked what they would do with a prediction, the answer was "we'd know".
Build the best damage predictor you can from everything in the tables, report its quality, and let the business decide what to do with it. A good prediction is useful to someone.
The best predictor uses carrier scan events and transit duration, which are known only after the shipment has left. It cannot inform packaging, which happens before, and it is worthless to the only team with an action and a budget.
- The best predictor uses carrier scan events and transit duration, which are known only after the shipment has left. It cannot inform packaging, which happens before, and it is worthless to the only team with an action and a budget.
- Its metric is overall ranking quality, but the packaging team can afford extra packaging on a fixed fraction of shipments; the number they need is precision within that fraction, and nobody computed it.
- It scores every shipment including the ones the fragile rule already handles; on those, extra packaging is already applied, so a "correct" high score changes nothing and the model's measured value is inflated by decisions already made.
- Six months later three teams each want a slightly different model, and the one that shipped serves none of them well because it was built for "knowing".
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Undetermined until the decision is named. "Damaged" is recorded at delivery inspection; a prediction of it could inform packaging at origin, routing mid-transit, insurance pricing at booking, or a claims reserve at month end — four decisions, four moments, four targets.
- Once the decision is fixed — extra packaging at origin for high-risk shipments — the target becomes "damage reported at delivery inspection", predicted at booking time, for shipments not yet packed.
- Booking records with origin, destination, carrier, declared contents and weight; a packaging log; carrier scan events; inspection reports at delivery with a damage flag and free-text notes.
- The packaging decision is currently a rule: fragile declared contents get extra packaging. The rule is applied by warehouse staff at packing time.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A decision has an owner, a moment, a set of available actions, a capacity and a cost for each mistake. Each of these fixes something about the model: the moment fixes the prediction time and therefore the legal features and the honest split; the actions fix the target — you predict the thing the action can change; the capacity fixes the threshold or the top-k; the costs fix the metric; the moment and volume together fix the inference mode.
- A model without a named decision has none of these fixed, so it is built with defaults — all features, a random split, a default metric, a threshold at one half, an online endpoint — and each default is right for some decision and wrong for the one that eventually arrives.
- The model is a component of the decision, not the reverse. Its value is the improvement in the decision's outcomes over the current way of making it, net of the cost of the actions it triggers — which is a number that cannot be computed without the decision.
One outcome, four decisions
The damage flag in the inspection report is one column, and it is the target of four different models depending on which decision it serves. The table makes the point the naive approach misses: the moment differs across the rows, and the moment decides which features exist. The best model for the last row is illegal for the first.
Ask which row you are building for before pulling data, because the answer decides most of what follows.
| Decision | Owner | Moment | Legal features | Action | Metric |
|---|---|---|---|---|---|
| Extra packaging | Warehouse | Booking | Booking fields only | Pack, at fixed capacity | Precision at capacity |
| Re-route | Operations | Mid-transit | Booking + scans so far | Change carrier leg | Recall at re-route budget |
| Price insurance | Commercial | Quote | Booking fields + customer history | Adjust premium | Calibration of P(damage) |
| Claims reserve | Finance | Month end | Everything, including scans | Set reserve | Calibrated expected count |
What the decision fixes
Take the packaging decision and follow what it determines. The moment is booking, so the features are the booking fields and nothing that happens after; the split must be time-based because shipments arrive in time; the action is binary and capacity-limited, so the output is a ranking and the operating point is the top-k; the costs are asymmetric, so the metric is precision within k and not accuracy; the volume is every booking in real time, so inference is online, in the booking path, under its latency budget.
None of these was a modelling choice. Each was a consequence of naming the decision, which is why naming it first is faster, not slower.
When does the packaging decision need the prediction, and how many are there?
when Packing follows booking within minutes and the booking system can wait for a score.
cost A serving path in the booking request's latency budget, with a fallback to the fragile rule when the model is unavailable.
when Bookings are packed the next morning from a picking list built overnight.
cost Same-day bookings are packed by the rule; the model influences only tomorrow's list.
when Most bookings are next-day, a minority are same-day and high-value.
cost Two serving paths, two feature computations and a skew test between them.
The model built for "knowing"
The naive damage model was measured honestly and looked good, and the reason it was worthless is not visible in its evaluation. Its features included the carrier scans; its population included the shipments the rule already packs; its metric weighted all shipments alike. Each was a default that a named decision would have replaced, and each inflated the offline number while removing the production value.
The device below records the gap. The cost line is the part worth remembering: the model that served the decision was cheaper to build and produced a smaller number.
Strong ranking of damaged against undamaged shipments on a random held-out set, all shipments, all features.
Not deployable: the top features are transit scans that do not exist at packing time. Re-scored with booking fields only, the ranking within the packing capacity was barely better than the fragile rule.
- 1The moment was never fixed, so the model learned from post-booking features that carry most of the signal and none of the actionability.
- 2The population included rule-covered shipments, which are easy to rank and already handled; removing them left the hard cases the model had little signal on.
- 3The metric was aggregate ranking, which rewards separating obvious cases; precision at packing capacity, on the uncovered population, is the number the decision needed.
How to build it
Most important first.
- Write the decision down with its owner, moment, actions, capacity and mistake costs, and get the owner to sign it before any data is pulled (Problem Formulation).
- Derive the target from the action: predict the outcome the action can change, at the moment the action is taken, for the population the action applies to (Target Definition).
- Derive the metric and threshold from the capacity and costs, and the inference mode from the moment and volume — a decision made at booking for every shipment is online; a nightly packing plan is batch (Choosing the Inference Mode).
- Exclude from the evaluation the cases the existing rule already decides, so the model is measured on the decisions it would actually change (The Rule Baseline, Beating the Baseline).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Damage rate among shipments that received extra packaging because of the model versus a control slice packed by the old rule, net of packaging cost. This is the decision's number and the only one that says the model paid for itself.
- Offline, precision at the packaging capacity on a time-based split over shipments the fragile rule does not already cover. This maps to the decision; aggregate ranking quality over all shipments does not.
- Do not measure the model on shipments the rule already packs. It gets those right by construction and the number says nothing about the decisions it changes.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The packaging decision is still made at booking, by the warehouse, with extra packaging as the available action and roughly the same per-shipment capacity.
- The cost of a wasted extra pack and the cost of an unpacked damaged shipment are still in the ratio the threshold was set for.
- The scores are consumed only by the decision they were built for; another team has not adopted them for a decision with a different moment.
- Offline: every feature is checked against the booking timestamp; the evaluation reports precision at capacity on a time-based split over the rule's uncovered population.
- Online: the fraction of scored shipments that actually receive extra packaging, against capacity; the damage rate on the model's picks against a control slice.
- Over time: a quarterly review with the decision owner covering moment, action, capacity and costs, and a check of who is reading the score table.
What can go wrong
- The decision is named and the capacity is not: the model ranks well, the warehouse packs everything above a threshold nobody costed, and the packaging budget is gone by the tenth of the month.
- The decision owner signs, then a second team adopts the scores for a different decision — insurance pricing — for which the target, moment and costs are all wrong, and the model's reputation goes with it.
- The decision changes shape — packaging moves from origin to a hub — and the moment moves with it; the features that were legal at booking are now a subset of what is known, and the model is quietly worse than it could be.
- Building for one decision means the model is deliberately less general; the three other teams who wanted damage predictions get nothing from it, and each would need its own formulation.
- Excluding the rule-covered shipments from evaluation makes the model's number look worse than an all-shipments number, which is harder to sell.
- A signed decision document is a commitment the business may not want to make before seeing whether the model is any good — a chicken-and-egg the formulation lab exists to break.
- "A good prediction is useful to someone." A prediction is useful at a moment, for an action, at a cost. The best damage predictor in the company was useless to every team that had an action, because it used features from after their moment.
- "We'll build the model first and find the decision later." The decision fixes the moment, and the moment fixes which features are legal. A model built with all the features cannot be retrofitted to an earlier moment; it has to be rebuilt.
- "The decision is obvious — reduce damage." That is a goal. A decision is a choice between actions made by someone at a moment, and "reduce damage" contains four of them with different targets.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the decision fixes the moment, target, metric, threshold and inference mode holds for every predictive system; exploratory analysis is the one setting with no decision, and it should not be called a model.
- CONTESTEDA serious position holds that insisting on a signed decision before any modelling kills exploration: often the business cannot know what decision a prediction could improve until it has seen that the prediction is possible, and a quick exploratory model is how formulation gets started. That is right — provided the exploratory number is never presented as the model's value, and the moment and features are re-derived from the decision before anything ships.
Where the depth lives
This domain teaches the model and hands the rest off by name.