The Software Delivery Lifecycle
Plan through learn as a chain of stages, each with defined inputs, outputs, automation, evidence and a way back.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What are the actual stages between an idea and a change users can depend on, and what does each stage owe the next?
Delivery is usually described as a diagram with arrows and no contract. When something goes wrong, nobody can say which stage was supposed to have caught it, what it was supposed to hand over, or how far back you have to go to undo it.
The lifecycle is the pipeline: someone opens a pull request, CI turns green, it merges, it deploys. The stages are whatever jobs are configured in the CI file.
The CI file describes what runs, not what each stage guarantees. A green pipeline means every configured job exited zero — which is a statement about the configuration, not about the change.
- The CI file describes what runs, not what each stage guarantees. A green pipeline means every configured job exited zero — which is a statement about the configuration, not about the change.
- Stages that produce no artifact quietly produce nothing. "We tested it" with no recorded result cannot be distinguished later from "the test job was skipped because the path filter did not match".
- The stages nobody automated — deciding to ship, verifying afterwards, learning from what happened — are exactly the ones that fail silently, because there is no red X when they are skipped.
- When an incident starts, the question is "what changed and how do we undo it". A pipeline modelled as jobs has no answer; a pipeline modelled as stages with outputs has one at every hop.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Each stage consumes a defined input and produces a defined output. Plan produces a described change; code produces a commit; build produces an artifact; release produces a decision; deploy produces running instances; verify produces a comparison against a baseline.
- The output of a stage is the only thing the next stage can trust. Anything a stage knew but did not write down is lost at the boundary — which is why "it worked on the branch" cannot be checked later.
- Evidence is separate from output. The output is the thing produced; the evidence is what proves the stage did its job. A build's output is an image; its evidence is a digest plus the commit it came from.
- Reversibility drops sharply at two points: the migration that changes data, and the release that reaches users. Before those, undoing is deleting a branch. After them, undoing is an operation with its own risk (Rollback: Only Useful If It Is Actually Safe).
- The chain runs forward but is read backwards. Every production question — why is this behaving differently, which change did it, can we go back — is a walk back up the stages looking for the last one whose evidence you trust.
Eleven stages, each with an output and an evidence
This is the whole module in one table. Each stage takes something, produces something, can fail in a characteristic way, and leaves behind evidence that it did its job. The evidence column is the one teams skip, and it is the one every incident needs.
Read the failure column as a checklist against your own pipeline. Most teams have three or four of these live at any time and have stopped noticing.
- 1Plan
Turns an intent into a described change with a stated shape, risk and rollback story.
fails by Scope decided without a rollback story, so reversibility is discovered at incident time.
evidence A written change description that names what it touches and how it comes back (Plan and Code).
- 2Code
Produces commits that implement the change, sized so cause and effect stay legible.
fails by One large commit bundling refactor, feature and migration — a bundle of suspects (Change Size: Why Small Changes Are Safer, and When They Are Not).
evidence Commits that each say what changed and why, joinable to the described change.
- 3Review
A second person checks intent, design, and the classes of defect humans are good at.
fails by Rubber-stamping a diff too large to read (Review as a Gate).
evidence An approval recorded against a specific commit, not against a branch that later moved.
- 4Build
Turns source plus pinned dependencies into an artifact.
fails by Unpinned inputs, so the same commit builds differently on different days (Reproducible Builds).
evidence A digest, plus a record of the commit and inputs that produced it (Build Provenance).
- 5Test
Executes checks against the change and the artifact.
fails by Flakiness normalised, so red is re-run rather than read (Flaky Tests).
evidence A retained result naming which checks ran against which commit.
- 6Package
Assembles the deployable form — image, bundle, chart — and publishes it.
fails by Rebuilt per environment, so what you tested is not what you ship (Build Once, Deploy Many).
evidence One immutable artifact in a registry, referenced by digest (Tags Versus Digests).
- 7Release
A decision that this artifact is a candidate for users, with its config and migrations.
fails by Conflated with deploy, so there is no decision point to record or refuse (Deployment Is Not Release).
evidence A release record naming artifact digest, config version and migration state (The Release Manifest).
- 8Deploy
Moves the released artifact onto infrastructure, progressively where possible.
fails by All instances at once, with no coexistence plan for old and new (Version Coexistence: N and N+1, in Both Directions).
evidence Instances reporting the new version and passing readiness (Probes: Readiness, Liveness and Startup).
- 9Verify
Compares the new version against a baseline on signals that reflect users.
fails by Skipped, because deploy reported success (A Successful Deploy Is Not Evidence of a Healthy System).
evidence Error rate and latency for the new version compared against the previous one, not against zero (Canary Analysis: Compared Against What?).
- 10Operate
Runs the system: alerts, on-call, routine maintenance, capacity.
fails by Manual work grows with the service until the team is fully consumed by it (Toil).
evidence Alerts that fire on user-visible symptoms and are acted on (Alert on Symptoms, Not on Causes).
- 11Learn
Turns what happened into a change to the system.
fails by A document instead of a merged change (Action Items That Change the System).
evidence A commit that exists because of an incident, traceable back to it.
The two joints that fail most often are release/deploy — collapsed together, so nothing is decided — and verify/operate — nobody checks, because the deploy tool went green.
Where reversibility ends
The cost of undoing a change is not constant across the lifecycle. It is near zero up to build, low until deploy, and then it steps sharply at two specific places: when data changes shape, and when users see the result.
That step is the reason plan-time rollback design matters. Once you are past it, "roll back" stops being an operation on your infrastructure and becomes an operation on your data and your users.
| You are at | Undoing means | Cost | What makes it worse |
|---|---|---|---|
| Plan / code | Delete the branch | None | Nothing; this is the cheapest place to change your mind |
| Review | Request changes | A conversation | A diff so large the reviewer cannot tell what to object to |
| Build / test | Discard the artifact | Wasted CI minutes | Nothing — build failures are the cheap kind |
| Package | Do not promote it | Registry storage | Mutable tags, so "the artifact" is ambiguous |
| Release | Do not deploy the candidate | Delay | No release record, so there is nothing to withhold |
| Deploy, pre-traffic | Redeploy the previous digest | Minutes | Not knowing which digest was previous |
| Deploy, serving traffic | Roll back and drain | Minutes plus in-flight requests | Connections not drained (Draining: Stopping Without Dropping) |
| After a schema change | Code back; data may not come back | Hours, sometimes never | A contract step already run (Destructive Migrations) |
| After users acted on it | System restored; user state is not | Support and communication | Money moved, emails sent, external systems notified |
The loop closes, or it is not a lifecycle
Drawn as a line, the lifecycle ends at deploy and implies that shipping is completion. Drawn as a loop, it makes the two neglected edges visible: verify feeding back into operate, and learn feeding back into plan.
Both neglected edges have the same shape. Something completed successfully, so it felt finished, and the step that would have made it useful was the one after it.
How to do it properly
Most important first.
- Name the output and the evidence for every stage you run. If a stage has neither, either give it one or stop pretending it is a gate.
- Carry one identifier — the commit SHA — through every stage, so that any stage can be joined to any other without asking a person (From Developer to Users).
- Decide the rollback strategy at plan time, not at incident time. The plan for a change that cannot be rolled back is a different plan (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Keep the stages that must stay human explicitly human: the release decision, and the judgement call during an incident. Automate everything mechanical between them (How to Automate Something).
- Make the pipeline the only path. A stage that can be bypassed by someone in a hurry is not a stage, it is a suggestion (Manual Production Changes).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a missing lifecycle — it shows up as every change being individually risky, because none of them can be traced or reversed systematically.
What can go wrong
- Stages that run but produce no retained artifact, so post-incident you cannot prove what was checked.
- Verify collapsed into deploy: the deployment tool reports success and everybody treats that as the change working (A Successful Deploy Is Not Evidence of a Healthy System).
- A migration stage coupled to the deploy stage such that rolling back the code leaves the schema ahead of it (A Migration and a Deploy Are One Event).
- A pipeline so slow that people work around it, which moves the real lifecycle outside the one you can observe (CI Is a Feedback System).
- Adding stages as a response to an incident until the pipeline is long, slow and mostly unread — more gates, not more safety.
- "More stages means safer delivery." Stages that nobody reads and nothing enforces add latency and no safety (Guardrails, Not Gates).
- "Green pipeline means the change is good." It means the configured checks passed on the merge base they ran against. It is a necessary condition, not a sufficient one.
- "Deploy is the last stage." Deploy is the middle. Verify, operate and learn are where most of the value and most of the neglect live.
- "The lifecycle is a straight line." It is a loop; the output of learn is the input of plan (The Production Loop).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For the change currently in production you can name: the commit, the artifact digest, the release record, the deploy time and the verification result.
- Every stage has a retained output you can point at weeks later, not a log line that has rotated away.
- Time from merge to production is measured, and the measurement is boring enough that nobody argues about it.
- The last rollback took minutes and nobody had to reconstruct which artifact to go back to.
- Before build: delete the branch. Cost is zero.
- After build, before release: mark the artifact as not a candidate. The artifact stays in the registry; nothing has moved.
- After deploy, before schema change: redeploy the previously released artifact by digest (Rollback: Only Useful If It Is Actually Safe).
- After a data-affecting migration: the code rollback is easy and the data rollback may not exist. This is why expand/migrate/contract exists (Expand, Migrate, Contract).
- After users have acted on the change: rollback restores the system, not the users' expectations. Communication becomes part of the recovery.
- Automate: build, test, package, publish, promote, deploy, health check, deploy annotation, and the mechanical parts of verification.
- Keep human: the decision that a candidate becomes a release, approval of destructive migrations, and the roll-back-or-forward call during an incident (The Automation Trap).
- Automate the recording, always. A human decision should still leave a machine-readable record of who decided and on what artifact (The Audit Trail).
- Explicit stage contracts cost engineering time and add pipeline surface. On a service that changes twice a year, the whole apparatus is more expensive than the incidents it prevents.
- Every gate adds latency between writing a change and learning whether it worked, and longer feedback loops make changes larger and riskier — the exact thing the gates exist to prevent.
- Retained evidence costs storage and retention policy. Retaining nothing is cheaper right up until the first serious incident.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe stage sequence holds for any software delivered to users, from a single container on one VM to a fleet. What changes is how many stages are automated and how much evidence is retained, not which stages exist.
- ORG-SPECIFICWhere the human approval sits is a policy choice. A regulated release train puts it before deploy and records it for audit; a consumer web team may put it nowhere and rely on canaries and flags instead. Both are coherent; copying one into the other's constraints is not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — what each class of test actually establishes, and why passing tests is evidence about the checks rather than about the change.