ReleaseGENERALTOOL-SPECIFICSCALE-SPECIFIC
Deployment Is Not Release
Deployment means code reaches an environment. Release means functionality becomes available to users. Conflating them makes both riskier than either needs to be.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
What is the difference between deploying a change and releasing it, and why does it matter operationally?
The problem
Two decisions with completely different risk profiles — "is this artifact safe to run" and "should users get this behaviour" — are made simultaneously by default, so neither can be made carefully and neither can be reversed independently.
What teams do first
Deploying is releasing. When the code is on the servers, the feature is out. If you do not want users to see something, do not deploy it yet.
How it breaks
It couples an engineering decision to a product decision. Shipping a feature on a specific date now requires a deploy on that date, which is the least flexible way to schedule anything.
How it breaks in production
- It couples an engineering decision to a product decision. Shipping a feature on a specific date now requires a deploy on that date, which is the least flexible way to schedule anything.
- It makes the branch the release mechanism. Work that is not ready to be seen stays unmerged, so branches live longer, diverge further and merge worse (Long-Lived Branches).
- It makes every release big. Batching everything until the feature is complete means the deploy contains many changes at once, and a problem afterwards has many suspects (Change Correlation).
- It makes rollback the only lever. If a feature misbehaves, the only response is to redeploy a previous artifact — which also removes every unrelated change that shipped with it.
- It removes any ability to expose functionality progressively. There is no "10% of users" if the only control is which bytes are running (Progressive Delivery: Exposure as a Dial).
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Deployment is a change to what is running: an artifact reaches an environment, becomes healthy, and serves traffic. Its risk is operational — does it start, does it stay up, does it serve correctly (A Successful Deploy Is Not Evidence of a Healthy System).
- Release is a change to what users can do: functionality becomes available to some population. Its risk is product and business — is this right, is it wanted, is it correct for this customer segment.
- They are separable because the code deciding whether a behaviour is active can be deployed long before the decision to activate it. A feature flag is the switch that makes the two independent (Feature Flags: Deploy Is Not Release).
- Once separated, each gets its own rate, its own approval, its own blast radius and its own rollback. Deploys become frequent, small and boring; releases become deliberate, gradual and reversible without a deploy.
- The reversal properties differ sharply, and this is the practical payoff: rolling back a deploy takes as long as a rollout, while turning off a release takes as long as a config propagation.
- This also changes what "done" means. Code merged and deployed but not released is finished engineering work sitting safely in production, which is a much better place for it than a branch.
Two changes, two risk profiles
The clearest way to see the distinction is to ask, of any change, who is harmed if it is wrong and what makes it stop. The answers are different in every row.
| Deployment | Release | |
|---|---|---|
| What changes | Which bytes are running | What users can do |
| Risk is | Operational — startup, health, correctness under load | Product and business — is this right, for whom |
| Decided by | The team that operates the service | Whoever owns the product decision |
| Reversed by | Deploying the previous artifact | Changing a flag or cohort |
| Reversal takes | A full rollout | A config propagation |
| Reversal side effects | Removes every other change in that artifact | None — only this behaviour |
| Natural frequency | As often as changes are ready | When the business wants the behaviour live |
| Recorded as | Artifact digest, commit, config version (The Release Manifest) | Behaviour, population, decider, time (The Audit Trail) |
The flag is the seam
Structurally, separating them means inserting one decision point between "the code is running" and "the behaviour happens". Everything else follows from that single seam.
- Everything left of the flag is a deployment concern: does it build, does it start, does it stay healthy, does it serve at the same error rate as before.
- Everything right of the flag is a release concern: who gets this, does it do the right thing for them, is the business effect what was expected.
- The default must be off. A flag that defaults on when the flag service is unreachable turns a flag outage into an unplanned release (How Networks Fail in Production).
- Both sides need a record. The commonest gap is a release with no record at all, because "nothing was deployed" (Deploys on the Same Timeline as the Symptom).
UserLLMAgentToolDataDecisionHumanGuardrail
One week, separated
GENERALIndependent of platform and flag vendor. The only variable is the propagation time at 10:36, which is a property of the flag system and worth knowing before you need it.
The value shows up when something goes wrong. Follow what is available at each point, and what would have been available if the two had shipped together.
A feature deployed Tuesday, released Thursday, withdrawn Thursday
- Tue 10:00changeFeature code merges to trunk behind a flag defaulting to off, and deploys with eleven unrelated changes.
- Tue 10:20signalDeploy verified: new version healthy, error rate matched to the previous version. The feature is in production and doing nothing.
- WedchangeFour more deploys, all unrelated. The flag is still off. None of them is blocked by an unfinished feature.
- Thu 09:00actionRelease begins: flag on for internal users. Not a deploy — no artifact changed.
- Thu 10:00actionFlag ramped to 5% of external traffic. Exposed cohort compared against the unexposed one (Canary Analysis: Compared Against What?).
- Thu 10:35signalError rate for the exposed cohort rises. The unexposed cohort is unaffected, which localises the cause immediately.
- Thu 10:36recoveryFlag off. Impact ends in seconds. No rollout, and the four unrelated deploys since Tuesday stay in production.
- Thu 10:40actionInvestigation proceeds with the code still deployed, still reproducible behind an internal-only flag (Production Debugging).
Had deployment and release been one event, Thursday 10:36 would have been a rollback: a full rollout, removing sixteen unrelated changes, taking minutes rather than seconds, and leaving nothing in production to debug against.
changesignalactionrecovery
How to do it properly
Most important first.
- Deploy continuously; release deliberately. Merge and deploy incomplete work behind a flag that defaults to off (Trunk-Based Development).
- Give every release a control that is not a deploy: a flag, a cohort, a configuration value. The control is what makes a release reversible.
- Record them separately. A deploy record names an artifact; a release record names a behaviour, a population and who decided (The Release Manifest).
- Ramp rather than flip. A release to 1%, then 10%, then everyone is three decisions with three chances to stop (Canary Analysis: Compared Against What?).
- Name the owner of each. The deploy is owned by the team that runs the service; the release is often owned by someone else entirely, and that is fine as long as it is explicit.
- Remove flags after the release is complete. A flag that has been on for everyone for months is a dead branch in the code with a live switch attached.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongOne percent
One testEveryone
What contains it
A release, done properly, is contained by the cohort it is exposed to — which is the entire point of separating it. The deploy underneath it is a different question with a different, usually larger, radius.
What can go wrong
Failure modes, including of the mitigation
- Flags that never get cleaned up, so the code accumulates conditionals and the combination of flag states is untested and untestable.
- A flag whose off path stops working, because nobody has exercised it since the release completed — so the emergency lever is broken exactly when it is needed.
- Flag changes made with no record, so a behaviour change in production has no corresponding entry anywhere and the incident starts with "nothing was deployed" (The Audit Trail).
- Using flags to hide an incompatible schema change, which does not work: the database does not have a flag (A Migration and a Deploy Are One Event).
- Treating the flag system as low-risk infrastructure. It is in the request path of every service that reads it, and its outage is a production outage (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Ramping a release and a deploy simultaneously, so when the metrics move you cannot tell which change caused it.
Misreads this invites
- "Feature flags mean we do not need canaries." Flags control who sees a behaviour; canaries control how many instances run new code. They answer different questions and you usually want both (Canary: One Percent, Then Five, Then Watch).
- "If it is behind a flag it is safe to deploy." The code is in the artifact either way. A flag guards the behaviour, not the process — a crash on startup happens whether or not the flag is on.
- "We release every deploy, so we are doing continuous delivery." You may be. The point of the distinction is that you *can* choose, not that you must always separate them.
- "Flags are a product feature." They are production infrastructure with an operational owner, a blast radius and a rollback story of their own.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- Deploy frequency and release frequency measured separately, and visibly different numbers. If they are identical, they are not separated.
- For a deploy: the new version is serving, healthy, and its error rate matches the previous version's (A Successful Deploy Is Not Evidence of a Healthy System).
- For a release: the exposed cohort shows the intended behaviour and the intended business effect, compared against the unexposed cohort.
- A flag has been turned off in production at least once deliberately, and the off path was verified to work.
- During an incident, you can say which of the two happened most recently, and when — from a record rather than from memory.
How you get back
- A release is rolled back by changing the flag, which takes as long as your flag propagation and needs no rollout.
- A deploy is rolled back by putting the previous artifact back, which takes as long as a rollout and removes every change in it (Rollback: Only Useful If It Is Actually Safe).
- The separation is what makes the cheap option available. Without it, every product problem is answered with an infrastructure operation.
- Neither helps with a schema change. That is a third thing with its own rules, and flags do not cover it (Expand, Migrate, Contract).
What to automate, and what stays human
- Automate deployment fully. It is repeatable, well-defined and benefits from being identical every time.
- Automate release *mechanics* — cohort assignment, ramp schedules, automatic rollback on a metric breach — but not the release *decision*.
- Keep human: who a release is exposed to, and when. That is a product decision with a business blast radius, and automating it means automating judgement (The Automation Trap).
What this costs
- Flags add branching to the code and to the test matrix, and every flag is a conditional someone has to reason about later.
- A flag system is another production dependency in the hot path, with its own availability requirement and its own failure modes.
- Deploying unreleased code means production contains behaviour nobody is exercising, which can hide bugs until the release.
- The discipline of cleaning up flags is unglamorous, has no deadline, and is therefore the part that slips.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe distinction holds on any platform. What differs is the mechanism for the release half — a flag service, a config value, a routing rule, an entitlement in the product itself.
- TOOL-SPECIFICFlag systems differ in ways that matter operationally: whether evaluation is local (SDK with a cached ruleset, so a flag-service outage is survivable) or remote (a call per evaluation, so the flag service is in your critical path), and how long propagation takes after a change. Both properties change your rollback time.
- SCALE-SPECIFICBelow a certain size, a config value and a redeploy is a perfectly good release control and a flag platform is overhead. The distinction still applies; only its implementation gets cheaper.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Observability & Performancedeployment-markers
Domains that do not exist yet
- — System Design — exposure control and cohorting as a product architecture concern, not only an operational one.