The Deployment Pipeline
The path from commit to production as a designed system: ordered stages, each with an input, a verdict, evidence, and a defined behaviour on failure.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the pipeline actually for, beyond running the build and the tests?
A pipeline assembled by accretion — a job added per problem, in the order the problems arrived — provides feedback in the wrong order, produces evidence nobody uses, and is trusted far beyond what it actually checks.
The pipeline runs the tests and then deploys. Add jobs as needed. If it is green, we are good.
Order determines feedback latency. A pipeline that runs a twenty-minute integration suite before a linter tells you about a formatting error twenty minutes late, every time (Designing the Pipeline).
- Order determines feedback latency. A pipeline that runs a twenty-minute integration suite before a linter tells you about a formatting error twenty minutes late, every time (Designing the Pipeline).
- "Green" is only meaningful if you can say what green asserts. In most accreted pipelines nobody can, so green becomes a ritual rather than a claim.
- Stages that produce no verdict — a scan whose output nobody reads, a test whose failure is routinely ignored — cost time and provide nothing (Scanning, and Why a Finding Is Not a Risk).
- Rebuilding per environment means the artifact tested is not the artifact deployed, which invalidates every earlier stage (Build Once, Deploy Many).
- A pipeline with no defined failure behaviour leaves partial state: an artifact pushed but not recorded, a migration applied but no deploy, a half-promoted release.
- The pipeline becomes production infrastructure with production credentials and no owner (CI Security).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A pipeline is a sequence of verdicts about one artifact, ordered so that the cheapest evidence that could disqualify it comes first, and so that each stage's evidence is preserved for the next one.
- Three properties make it a pipeline rather than a list of jobs: one artifact flows through it, each stage produces a verdict rather than only output, and the order is by information value per unit of time.
- The artifact is built once, early, and identified by digest. Everything after that stage tests, scans, promotes or deploys that specific digest — never a rebuild (Tags Versus Digests).
- Stages divide into gates (which can stop the artifact) and observations (which cannot). Both are legitimate; conflating them is not. A scan that never blocks is an observation and should be labelled as one, so nobody believes it is protecting them.
- Environments are stages in the same pipeline, not separate pipelines. Promotion moves the same digest forward with different configuration (Promotion).
- The pipeline's own definition is code, is reviewed, and is deployed — and it has the highest privilege of anything in your delivery path (Policy as Code).
One artifact, a sequence of verdicts
The stages below are a common shape rather than a prescription. What matters is that each has an input, a verdict, evidence and a defined behaviour on failure — and that from the build stage onward, they all refer to the same digest.
- 1Static checks
Formatting, linting, type checks — the cheapest disqualifiers.
fails by Running after expensive stages, so trivial failures cost the full pipeline duration.
evidence Fails in under a minute when it fails.
- 2Unit tests
Fast behavioural checks on the change itself.
fails by Flakiness, which converts a gate into a re-run habit (Flaky Tests).
evidence Deterministic pass/fail; failures reproduce locally.
- 3Build
Produces the one artifact, identified by digest.
fails by Unpinned inputs, so the same commit produces different bytes (Reproducible Builds).
evidence A digest, recorded, and referenced by every later stage (Tags Versus Digests).
- 4Integration tests
Exercises the artifact against real dependencies.
fails by Testing a rebuild rather than the artifact, or against stubs that do not fail like the real thing.
evidence The tested digest matches the built digest.
- 5Security and supply chain
Scans dependencies, signs the artifact, records provenance.
fails by Producing findings nobody triages, so the stage is an observation everyone believes is a gate (Scanning, and Why a Finding Is Not a Risk).
evidence A signature and an SBOM attached to the digest (Signing and Verifying Artifacts).
- 6Promote to a lower environment
Deploys the same digest with different config.
fails by Rebuilding for the environment (Build Once, Deploy Many).
evidence Same digest, different config version, service healthy.
- 7Migration step
Applies schema changes once, in order, before the rollout.
fails by Running per-replica at startup, or applying a destructive change unattended (A Migration and a Deploy Are One Event).
evidence Schema version recorded and matching what the artifact expects.
- 8Progressive production rollout
Canary, compare, promote or roll back.
fails by Deploying to the whole fleet at once, so detection and full impact coincide (Canary: One Percent, Then Five, Then Watch).
evidence Canary verdict against a live baseline (Canary Analysis: Compared Against What?).
- 9Record
Writes the release manifest and annotates observability.
fails by Being skipped on the emergency path, so the incident release is the one with no record (The Release Manifest).
evidence A manifest and a deploy annotation exist for every production change (Deploys on the Same Timeline as the Symptom).
The migration step is placed deliberately: after lower-environment verification, before the production rollout, and as its own stage rather than inside the deploy. That position is what makes the compatibility window bounded and observable.
Gates and observations are different things
Every stage either can stop the artifact or cannot. Both are useful. The damage comes from stages whose category nobody has stated, because everyone assumes they are gates.
- A gate has a verdict that stops the artifact. It must be reliable, or it will be bypassed.
- An observation records something for later. It must be labelled, or it will be mistaken for protection.
- A stage that blocks intermittently is the worst of both: unreliable as a gate, and it trains people to re-run.
- The canary verdict is the last gate, and it is the only one that runs against real traffic (A Successful Deploy Is Not Evidence of a Healthy System).
Pipeline failures that are not test failures
These are the failures that make a pipeline untrustworthy without ever turning it red — which is why they persist for so long.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Rebuild in a later stage | Production behaves differently from the environment that passed | The deployed artifact was not the tested artifact | Build once; reference the digest everywhere after (Build Once, Deploy Many) |
| Long pipeline duration | Engineers batch changes; each failure has many suspects | Feedback latency changed behaviour, as it always does | Reorder by disqualifier cost and parallelise the slow independent stages (Parallelising CI) |
| Non-blocking stage that was meant to block | A known-bad artifact reaches production with a green pipeline | A stage marked advisory during an incident and never restored | Audit gate/observation status; make the category explicit in the definition |
| Emergency bypass used routinely | The changes with the least verification are the ones shipped under pressure | The normal path is slower than the urgency tolerates | Fix the normal path's latency; make the bypass loud, logged and reviewed (Manual Production Changes) |
| Broad pipeline credentials | A compromised pipeline job has production write access to everything | One credential covering every deployment target | Short-lived, per-target, workload-identity credentials (Workload Identity) |
| Partial run failure | An artifact in the registry that never passed the later stages | No compensation on failure after the publish step | Publish only after gates, or mark unpromoted artifacts explicitly (Promotion) |
How to do it properly
Most important first.
- Order stages by how quickly they can disqualify the artifact per minute of runtime: static checks, unit tests, build, integration tests, security scans, deploy to a lower environment, deploy to production progressively.
- Build once, then reference the digest at every subsequent stage. If a stage rebuilds, everything before it stops applying.
- Give every stage an explicit verdict and an explicit failure behaviour: block, warn, or record. Write it down where the pipeline is defined.
- Make the pipeline's definition part of the repository so a change to it is reviewed like a change to the service (Infrastructure as Code).
- Carry evidence forward: which commit, which digest, which config version, which checks passed, which migration ran (The Release Manifest).
- Design for resumption. A pipeline that must restart from the beginning after a transient failure encourages people to bypass it.
- Keep a documented path to production that does not require the pipeline, and rehearse it — because one day the pipeline will be the thing that is broken (Break-Glass Access).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The pipeline is the containment for everything else — it is where gates, canaries and approvals live. Its own failures are contained by very little, which is why a non-pipeline path to production has to exist and be rehearsed.
What can go wrong
- Slow feedback that changes behaviour: engineers batch changes to avoid the wait, which makes each pipeline run more expensive to diagnose (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Flaky stages that are routinely re-run, which converts a gate into a delay (Flaky Tests).
- A stage that has been failing and marked non-blocking "temporarily", indefinitely.
- Environment-specific rebuilds introduced for a good local reason, silently invalidating everything upstream.
- Pipeline credentials with broad, long-lived production access, because the pipeline needs to deploy and nobody scoped it further (Secrets in CI).
- The mitigation failing: a manual override path added for emergencies that becomes the normal path whenever the pipeline is slow (Manual Production Changes).
- Partial failure with no compensation — an artifact pushed to the registry from a run that later failed, so the registry contains builds that never passed (Artifact Retention).
- "Green means safe." Green means every gate's assertion held. Whether those assertions cover the risk is a separate question, and answering it honestly is the most valuable pipeline review you can do.
- "More stages means more safety." More stages means more time, and every stage that does not produce an acted-upon verdict is pure cost (What an Environment Is For makes the same argument about environments).
- "The pipeline is developer tooling." It has production credentials and it is the only sanctioned path to production. It is production infrastructure (The Delivery Chain as Attack Surface).
- "We can add the security scan later." Where the scan sits determines whether it can block. A scan after the deploy stage is a report, not a gate.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For any artifact in production, you can list every stage it passed and when — from a record, not from a CI web UI you have to click through.
- The digest in production appears in the build stage's output for the same run.
- Median pipeline duration is short enough that nobody batches to avoid it, and the number is tracked.
- Every stage can be classified as gate or observation, by someone who did not write it.
- A recent transient failure was resumed rather than restarted, or the restart was cheap enough not to matter.
- The pipeline's own changes roll back like any other code, provided the definition is versioned with the repository.
- A failed pipeline run should leave no partial state that a later run must clean up. Where it cannot — an artifact already pushed — the compensating action should be automatic (Artifact Retention).
- The deployment stage's rollback is the service's rollback, and the pipeline should be able to trigger it as a first-class operation rather than as "run the previous pipeline again" (Rollback: Only Useful If It Is Actually Safe).
- Automate the whole path. A pipeline with a manual step in the middle has the latency of a human and the reliability of a habit.
- Automate evidence capture as a side effect of each stage, so the release record assembles itself (The Release Manifest).
- Keep human: the design of the pipeline itself, which stages are gates, and the risk acceptance when a gate is bypassed (Change Management).
- Comprehensive stages cost wall-clock time on every change, and pipeline latency is a direct tax on the feedback loop (CI Is a Feedback System).
- Parallelising for speed costs compute and makes failures harder to read (Parallelising CI).
- Strict gates block bad changes and also block good changes when the gate is wrong, and the cost of a false positive is paid every time.
- A pipeline sophisticated enough to be worth this care is itself a system that needs an owner, a runbook and maintenance (Toil).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICCI systems differ in ways that change pipeline design: whether stages can pass artifacts by reference or must re-upload them, whether a run can resume from a failed stage, how caching is scoped, and how deployment credentials are issued to a job. Copying a pipeline shape between tools usually breaks on one of these.
- GENERALThe design principles — one artifact, verdict per stage, cheapest disqualifier first, evidence carried forward — hold regardless of tool.
- ORG-SPECIFICWhich stages are mandatory gates is a policy decision. A regulated environment may require a scan and an approval that a startup would run as observations, and both can be correct for their context.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.