ReleaseGENERALORG-SPECIFICSCALE-SPECIFIC

Continuous Delivery

The practice of keeping software in a state where any commit on the main line could be released — whether or not you choose to release it.

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What does "always in a releasable state" actually require, and how would we know we have it?

The problem

When releasing requires work — a stabilisation branch, a manual test pass, a hand-assembled artifact — releases get batched to amortise that work, and batching is what makes them dangerous.

What teams do first

We release every two weeks. Before each release we branch, freeze, test, fix what we find, and ship. The freeze is what makes it safe.

How it breaks

The freeze does not remove risk; it accumulates it. Two weeks of changes ship together, and when something breaks afterwards every one of them is a suspect (Change Correlation).

How it breaks in production
  • The freeze does not remove risk; it accumulates it. Two weeks of changes ship together, and when something breaks afterwards every one of them is a suspect (Change Correlation).
  • The stabilisation period is a signal that the main line was not releasable. Fixing it per release means fixing symptoms repeatedly rather than the property once.
  • Work completed on day one waits thirteen days for feedback, so the cost of a mistake is discovered long after the context is gone.
  • The release becomes an event with its own preparation, which means it needs scheduling, which means fewer of them, which makes each one bigger. The loop reinforces itself.
  • Emergency fixes have to bypass the whole process, so the least-tested path is the one used under the most pressure (Break-Glass Access).
CodeBuildTestArtifactReleaseDeployRunObserveOperateIncidentRecoverLearnImprove

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • Continuous delivery is a property of the main line, not a deployment frequency: at any commit, the software could go to production without additional work.
  • That property has to be established by evidence produced automatically, because a property that requires a person to verify is not continuously true.
  • The evidence chain is: every change is integrated into the main line frequently; automated checks run against it; the artifact is built once and promoted rather than rebuilt; deployment to production is a routine, rehearsed operation; and unreleased work is neutralised by flags rather than by branches (Build Once, Deploy Many).
  • The release decision remains human and deliberate. That is the whole distinction from continuous deployment: continuous delivery says you *could* release any commit; continuous deployment says every verified commit *does* go (Continuous Deployment).
  • The main line being releasable also changes the meaning of a red build: it is not "the build is broken", it is "we currently cannot release", which is a much sharper statement of the cost (CI Is a Feedback System).

What "releasable" has to mean to be checkable

The phrase is only useful if it decomposes into things a pipeline can assert. Each of these is a claim, and each has a way of being false while looking true.

The claims behind a releasable main line
  1. 1
    Integrated

    Every change is on the main line, not waiting on a branch.

    fails by Long-lived branches, or a merge queue deep enough that the main line lags reality (Long-Lived Branches).

    evidence Branch age distribution is short; the main line is what everyone builds on.

  2. 2
    Checked

    Automated checks assert the behaviour that matters.

    fails by Checks that pass regardless — flaky, tautological, or covering the wrong layer.

    evidence Recent production defects are traced to whether a check could have caught them; the honest answer changes the suite.

  3. 3
    Built once

    One artifact is produced and promoted unchanged.

    fails by Rebuilding per environment, so what you tested is not what you ship (Build Once, Deploy Many).

    evidence The digest in production matches the digest that passed the checks (Tags Versus Digests).

  4. 4
    Configurable

    Environment differences are configuration, not code paths.

    fails by Environment-specific branches in code, so the tested path is not the production path (Build-Time and Runtime Configuration).

    evidence Config is validated at startup and its version travels with the release (Validate at Startup, Fail Clearly).

  5. 5
    Deployable on demand

    Deployment is routine, rehearsed and needs no preparation.

    fails by A deploy runbook with manual steps, which decays between uses.

    evidence Deploys happen often enough that nobody prepares for one.

  6. 6
    Reversible

    The previous artifact can be restored, and has been.

    fails by A rollback path that exists on paper (Rollback: Only Useful If It Is Actually Safe).

    evidence A recent rollback that was uneventful.

  7. 7
    Neutralised

    Unfinished work is present but inert behind flags.

    fails by Half-finished behaviour reachable in production because the flag only covers the UI (Feature Flags: Deploy Is Not Release).

    evidence Flag defaults are off and the off path is exercised by the checks.

The property is the conjunction. Six of seven means the main line is releasable except when it is not, which is indistinguishable from the release-branch model with more ceremony.

Batching is the mechanism, not the cadence

The argument for frequent releases is not that shipping is good. It is that risk scales with the number of changes in a release and with how long ago each was written, and both of those are consequences of batching.

Same changes, two shapes
Batched fortnightly release
day 1-10   34 changes merge to develop
day 11     branch, freeze, stabilise
day 11-13  manual test pass; 6 fixes cherry-picked
day 14     release all 34 together
day 14+2h  error rate rises
           -> 34 suspects, 6 of them written 13 days ago
           -> rollback removes all 34
           -> the next release is now 3 days behind, so it is bigger
Continuously delivered
day 1-14   34 changes merge to trunk, each released on merge
           each release: 1 change, verified against a baseline
day 7      one release raises error rate
           -> 1 suspect, written 20 minutes ago, by someone still at their desk
           -> rollback removes 1 change
           -> nothing else is delayed

Not because the second team is more careful — the changes are identical. Attribution cost scales with batch size, and the age of a change determines how much context is still available when it fails. Batching maximises both.

How often to release, given that you can

ORG-SPECIFICThere is no correct answer here, and the common failure is copying a cadence from a company with a different product shape. What generalises is that the cadence should be a decision you revisit, not a constraint your tooling imposes.

Continuous delivery makes release cadence a choice rather than a constraint. That choice is genuine, and different answers are right for different products.

Choosing a release cadence once the main line is always releasable

You can release any commit. How often do you?

On every merge, automatically

when Strong automated verification, fast rollback, and a blast radius you can contain progressively.

cost This is continuous deployment, and it demands verification good enough to be the last gate (Continuous Deployment).

On demand, several times a day

when You want a human decision point but no batching. The common middle, and usually the right one.

cost Someone must be available to decide, and the decision can become a rubber stamp that adds delay without adding scrutiny.

On a daily or weekly train

when Multiple services must ship together, or downstream consumers need predictability.

cost Reintroduces batching in proportion to the interval, and creates pressure to make the train.

On a business schedule

when Releases are coordinated with marketing, contracts or regulatory windows.

cost Only sane when deployment and release are separated — deploy continuously, release on the schedule (Deployment Is Not Release).

When the customer accepts it

when Installed or on-premise software where you do not control the runtime.

cost Many versions in the field at once, which changes the whole support and compatibility model (Running Two API Versions in One Service).

How to do it properly

Most important first.

  • Integrate into the main line frequently — at least daily per engineer — so the main line reflects reality rather than a merge queue (Trunk-Based Development).
  • Make the automated checks the definition of releasable, and make them fast enough that they run on every change (Designing the Pipeline).
  • Build the artifact once and promote the same digest through environments. An artifact rebuilt per environment is not the thing you tested (Promotion).
  • Keep unfinished work in the main line behind flags, not on branches (Feature Flags: Deploy Is Not Release).
  • Rehearse the deploy constantly. Deploying to production regularly is what makes deploying to production unremarkable.
  • Treat a red main line as an outage of your ability to release, and fix it before starting new work (Required Checks).
  • Eliminate flaky tests aggressively — a check people re-run until it passes has stopped being evidence (Flaky Tests).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

The practice is itself a containment mechanism: small changes released frequently have a smaller radius each and a recent, well-understood rollback target. Losing the practice raises the radius of every subsequent release.

What can go wrong

Failure modes, including of the mitigation
  • The pipeline is green and the software is not actually releasable, because the checks do not cover the thing that breaks. Green becomes a habit rather than a signal.
  • Flaky tests degrade the meaning of green until "re-run it" is the standard response, at which point the releasable property is unverified.
  • A "releasable" main line that has never actually been released from an arbitrary commit, so the property is theoretical.
  • Long-lived branches reappear under new names — a develop branch, a release branch, an integration branch — reintroducing batching while everyone believes they are doing CD (Git Workflows).
  • Database changes not held to the same standard, so the main line is releasable except when it contains a migration — which is exactly when it matters (A Migration and a Deploy Are One Event).
  • The mitigation failing: a heavyweight approval added to protect the releasable main line, which reintroduces batching by making each release expensive again.
Misreads this invites
  • "Continuous delivery means deploying continuously." It means being *able* to. The deploy cadence is a separate choice, and it is the one continuous deployment makes for you (Continuous Deployment).
  • "We have a CI pipeline, so we do continuous delivery." CI establishes that changes integrate. CD establishes that the result is releasable, which is a stronger claim requiring more evidence (Continuous Integration).
  • "CD requires microservices." It requires a main line you can release. A monolith with good checks delivers continuously; a fleet of services with a coordinated release train does not.
  • "We cannot do this, we are regulated." Regulation usually constrains who approves a release and what evidence is retained. Both are compatible with an always-releasable main line, and automated evidence is often easier to audit than a manual pass (Change Management).

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • You can pick an arbitrary recent commit on the main line and release it, and you have done so — not as a drill, as normal operation.
  • Time from merge to a production-ready artifact is measured, and is short enough that nobody batches to avoid it.
  • The main line is green nearly all the time, and when it is red it is fixed as the first priority rather than worked around.
  • The number of long-lived branches is close to zero, and the ones that exist have a stated end date.
  • Deploys are frequent enough that no single one is an event requiring preparation (A Successful Deploy Is Not Evidence of a Healthy System).
How you get back
  • Continuous delivery is a practice rather than a change, but adopting it is reversible in one direction only: it is easy to stop and hard to restart, because the tests and the flag discipline decay quickly.
  • The operational rollback story it enables is the real payoff: small, frequent releases mean the previous artifact is recent, well understood and close in behaviour to the current one (Rollback: Only Useful If It Is Actually Safe).
  • If the practice is failing — green builds that are not releasable — the correct response is to strengthen the evidence, not to add a manual gate back on top of it.
What to automate, and what stays human
  • Automate everything that establishes releasability: integration, checks, build, artifact promotion, environment provisioning, deployment.
  • Automate the enforcement of the property: protected main line, required checks, no merges on red (Protected Branches).
  • Keep human: the release decision itself, and the judgement about what evidence a particular change needs beyond the standard set (Review as a Gate).
What this costs
  • The automated check suite is a substantial and permanent investment, and it is the load-bearing part — a weak suite makes the whole practice a formality.
  • Flag discipline costs code complexity and a test matrix that grows with the number of live flags.
  • Frequent small releases mean more release events to observe, which needs deploy annotations and per-version metrics to stay legible (Deploys on the Same Timeline as the Symptom).
  • For software with genuine batch delivery constraints — installed clients, firmware, regulated release windows — the practice applies to the artifact's readiness, not to its delivery cadence, and the benefit is smaller.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • GENERALThe releasable-main-line property is independent of language, platform and architecture. What varies is the cost of establishing it, which is dominated by how good the automated checks are.
  • ORG-SPECIFICWho may authorise a release, what evidence must be retained and whether a change advisory step is mandatory are organisational and sometimes regulatory choices. None of them require batching — they require the evidence to be produced and kept, which automation does better than a manual pass.
  • SCALE-SPECIFICA single team on a single deployable gets most of this nearly free. Coordinating a releasable state across many services that must ship together is a much harder problem, and the usual answer is to remove the must-ship-together constraint rather than to coordinate it (Microservices).

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Observability & Performanceperformance-regression-detection
Domains that do not exist yet
  • Testing & Reliability Engineering — the automated check suite is the load-bearing component of this practice, and its quality sets the ceiling on everything above it.