The MLOps Pipeline
Data → Validation → Training → Evaluation → Artifact → Registry → Deployment → Monitoring. Each stage has a way of failing that the next stage cannot see.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A model went from raw data to production through eight stages. Which stage let the bad model through, and what would have stopped it there?
We retrained our delivery-time model last Thursday and shipped it Friday. On Monday customer support was flooded: estimates were off by hours for a whole region. The pipeline ran green end to end. We want to know which step should have caught it, because "the pipeline was green" is not an answer we can give again.
A pipeline is a sequence of scripts: extract, train, evaluate, upload, deploy. Each one runs if the previous one exited zero. Green means good.
The extract exited zero with a region's data mostly missing. There was no validation stage, so training happily fitted on the rows that survived — old rows, from before a road closure that now dominates the region's delivery times.
- The extract exited zero with a region's data mostly missing. There was no validation stage, so training happily fitted on the rows that survived — old rows, from before a road closure that now dominates the region's delivery times.
- Evaluation used a random split of the same shrunken data, so the region's held-out rows were from the same stale period and the metric looked fine. The overall metric moved very little because the region is a small share of orders (Evaluation Slices).
- The artifact was uploaded and the deployment stage swapped it in for all traffic. Monitoring watched latency and error rate, both perfect. The first signal was support tickets, three days later.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the delivery time of an order at checkout, in minutes. The label is the actual delivery timestamp minus the order timestamp, which is known once the order is delivered.
- The surrounding system's target is that a retrained model reaches production only when each stage has proven the thing it is responsible for, and that a regression can be localised to the stage that missed it.
- Orders joined to courier events and weather. Training data is a nightly warehouse extract; one example is one delivered order with the features as they were at checkout time, reconstructed by a point-in-time join.
- Last week a courier-tracking vendor changed its event schema for one region. Delivery timestamps for that region arrived as nulls, which the extract silently dropped. The training set for the region shrank to a few hundred rows, all of them from the days before the change.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Each stage of the pipeline produces something the next consumes, and each has a characteristic way of failing that the next stage cannot detect from its input alone. Training cannot tell the data is incomplete; evaluation cannot tell the split is wrong; deployment cannot tell the metric was computed on the wrong slice. So each stage needs its own check, before it hands off.
- The stages are: Data (assemble the training set), Validation (check it against expectations — volume, schema, distributions, label completeness), Training (fit, with a smoke test that it learned anything), Evaluation (compare against the incumbent on the right split and the right slices), Artifact (bundle the model with its preprocessing and contract), Registry (record and stage it), Deployment (roll it out gradually with a fallback), Monitoring (watch features, predictions and outcomes).
- The flagship at
/ml/pipelinewalks the same stages and shows which of the sixteen failure classes enters at each one. The pipeline here is the operational version: the same stages as jobs with gates between them.
Eight stages, eight ways to be silently wrong
The pipeline is a chain of hand-offs. Each stage trusts its input and produces an output the next stage will trust. That trust is the vulnerability: a stage cannot check what it cannot see, and each stage sees only its own input.
So the device below pairs every stage with how it fails when the previous one handed it something wrong, and the check that belongs at the hand-off. The delivery-time incident entered at Data and was caught at Monitoring, having passed five gates that did not exist.
- 1Data
Assemble the training set from sources with a point-in-time join
fails by A source silently loses rows or shifts meaning; the extract exits zero with a region missing
- 2Validation
Check volume, schema, null rates, label completeness and distributions against a stored profile
fails by The profile is stale and the gate is disabled; or the check is per-table, not per-segment
- 3Training
Fit the model on the validated set with a smoke test that loss fell and predictions vary
fails by Training converges on a degenerate set — fewer rows, one class, a constant feature — and reports success
- 4Evaluation
Compare against the incumbent on a time-respecting split and on named slices
fails by A random split hides temporal collapse; an aggregate metric hides a slice regression
- 5Artifact
Bundle weights, preprocessing and feature contract; record lineage
fails by The artifact ships without its preprocessing, and serving reimplements it differently
- 6Registry
Store, version and stage the artifact with its evaluation record
fails by Promotion is a manual flag anyone can set; the evaluation record is not required
- 7Deployment
Roll out gradually with a canary, a shadow or both, with a fallback ready
fails by Full swap on success; the canary window misses the affected segment
- 8Monitoring
Watch feature, prediction and outcome distributions, joined to model version
fails by Watches latency and errors only; outcomes arrive late; the alert has no owner
The flagship at /ml/pipeline shows the sixteen failure classes at the stage each enters. The operational lesson is that the cheapest place to catch a failure is the first gate after it enters — every later gate is more expensive and less specific.
The metric was computed on the wrong data
Evaluation is supposed to be the stage that stops a bad model. It failed here for a reason worth being precise about: it was given a training set that already excluded the current period for one region, and it split that set randomly. Every held-out row for the region was also stale. The metric was an honest measurement of a model on data that no longer resembled production.
This is the general pattern of the offline/online gap in pipeline form — the number is right, the thing it was computed on is wrong, and the stage that could have noticed was upstream.
Overall error on the held-out split marginally better than the incumbent; the affected region's slice was not evaluated separately.
Estimates off by hours for the region from Friday; support tickets Monday; a rollback Tuesday once someone joined the tickets to the model version.
- 1The extract dropped most recent rows for the region, so both training and validation for that region were from before a road closure that now dominates delivery times.
- 2The random split placed stale rows on both sides, so the held-out set could not reveal that the region had changed.
- 3The region is a small share of overall traffic, so its regression barely moved the aggregate metric the gate looked at.
What the validation gate checks, concretely
The validation stage is where this incident was cheapest to catch, and it is the stage most often absent, because it produces nothing — it only says no. Its job is to compare the assembled training set against a profile of what the training set usually looks like: row counts per segment, null rates per column, label completeness, and the ranges of key features.
The profile has to be maintained. A profile computed once and never refreshed either goes stale and blocks everything, or gets disabled. The practical shape is a profile refreshed from each passing run with a tolerance band, and a review when the band itself needs to move.
The assembled training set has the volume, completeness and distribution per segment that the model will meet in production.
holds when Sources are stable, the extract is validated per segment against a maintained profile, and label completeness is asserted.
breaks when A source changes schema for a subset; an upstream filter silently tightens; a segment's labels start arriving late and the extract excludes them.
respond Block training, fix the extract, and check whether the incumbent model was trained on the same broken data — it may already be degraded.
1def validate_training_set(df, profile, min_ratio=0.6):2 # profile: per-segment row counts and per-column null rates from recent passing runs3 problems = []4 counts = df.groupby("region").size()5 for region, expected in profile["rows_per_region"].items():6 got = int(counts.get(region, 0))7 if got < expected * min_ratio:8 problems.append(f"{region}: {got} rows, expected ~{expected}")9 for col, expected_null in profile["null_rate"].items():10 got_null = df[col].isna().mean()11 if got_null > expected_null + 0.05:12 problems.append(f"{col}: null rate {got_null:.2f}, expected ~{expected_null:.2f}")13 if df["label_minutes"].isna().any():14 problems.append("label_minutes has nulls; the extract should have excluded them")15 return problems # non-empty blocks training and writes to the run recordThe check is per segment. A whole-table row count would have passed here, because one small region losing most of its rows barely moves the total. Segment is the dimension the business cares about, so it is the dimension the gate needs.
How to build it
Most important first.
- Put a validation gate between data and training that checks row counts per segment, null rates per column and label completeness against a stored profile (Data & Feature Tests). This gate alone catches the regional collapse.
- Evaluate against the incumbent model on a time-based split and on named slices, and block promotion if any critical slice regresses (Champion / Challenger, Time-Based Split).
- Deploy through a canary with an outcome-aware check where labels are fast, and a prediction-distribution check where they are slow (Canary Rollout, Prediction Drift).
- Make every stage write its check result to the run record, so "which stage let it through" is a query rather than an investigation (Model Lineage).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per-stage gate outcomes per run: did validation pass, did evaluation pass on every slice, did the canary pass. These localise the failure.
- Time from a bad input to detection, by stage. The goal is that it shrinks toward the validation stage, where it is cheapest.
- A green pipeline status is a fact about exit codes. It maps to nothing about model quality unless the gates are the ones described here.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The validation profile describes the current data — it is refreshed on a schedule and its refresh is itself reviewed, so the gate is neither stale nor rubber-stamped.
- The evaluation slices include every segment whose failure would be a product incident, and the split respects time.
- Every stage writes its result to the run record, so a failure can be attributed to a stage after the fact.
- Offline: replay last quarter's incidents through the pipeline with the gates in place. Each one should be caught at a named stage; an incident no gate catches is a missing gate.
- Online: inject a known-bad input on purpose — drop a region from the extract in a staging run — and confirm validation blocks it.
- Over time: track the stage at which each real failure was caught. If the answer keeps being "monitoring", the earlier gates are not doing their job.
What can go wrong
- The validation profile is stale: the stored expectations were computed a year ago and every run now fails, so the gate is disabled "temporarily".
- Slice evaluation exists but the slices were chosen by geography, and the failure is by device type. Slices catch what they were defined to catch.
- The canary passes because the region in question gets little traffic in the canary window, and full rollout follows.
- Eight gated stages take longer than five scripts, and every gate needs an owner who keeps its expectations current.
- Slice evaluation multiplies the evaluation work by the number of slices and makes the promotion decision multi-dimensional, which invites arguments.
- A canary with outcome checks needs labels that arrive within the canary window; where they do not, the canary can only check distributions.
- "The pipeline was green, so the model was fine." Green means every stage exited zero. Without validation and slice gates, a pipeline is green when it has trained on garbage.
- "Add more monitoring." Monitoring is the last stage; it caught the failure three days late. The fix belongs at validation, where the input was already wrong.
- "Our data pipeline already has quality checks." Those check the warehouse tables. The training extract is a different artifact with its own way of losing rows.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvery ML system moves through these stages whether they are scripts or a managed workflow, and each stage has the same characteristic blind spot regardless of model family or domain.
- SCALE-SPECIFICA team with one model retrained monthly can run the gates by hand from a checklist; a team with fifty pipelines needs the gates automated and their results stored, because nobody will read fifty checklists.
- SIMPLIFIEDThe eight-stage pipeline is a teaching shape; real pipelines fork for multiple models, loop back from monitoring to data, and merge stages. Any numbers in the illustrations describe the shape of the failure, not a measured system.
Where the depth lives
This domain teaches the model and hands the rest off by name.