Progressive Delivery: Exposure as a Dial
Combining a rollout strategy, an automated comparison and a release control into one idea — exposure that increases only while evidence supports it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does it look like when the strategies in this module stop being alternatives and become one system?
Each strategy on its own controls one variable. A canary bounds how many users see a bad artifact but cannot bound a feature that is wrong for a specific customer segment; a flag controls a feature but not the dependency upgrade shipped in the same artifact. Real changes need both, gated by something that decides whether to continue.
We already do canary deploys and we have feature flags. That is progressive delivery — the name is just marketing for things we do.
Having the mechanisms is not the same as having the gate. If a human decides "looks fine" at each step, the rollout proceeds at the speed and rigour of whoever is watching, which varies (Canary Analysis: Compared Against What?).
- Having the mechanisms is not the same as having the gate. If a human decides "looks fine" at each step, the rollout proceeds at the speed and rigour of whoever is watching, which varies (Canary Analysis: Compared Against What?).
- Without automatic abort, the value of a small step is mostly lost: exposure stays small only until someone notices, and noticing is the slow part.
- Canary and flags are usually operated by different people with different tooling, so during an incident nobody can say whether the cause is the artifact or a targeting rule that changed an hour ago.
- Ramp schedules chosen for convenience — a step every five minutes — produce windows too short for the failure classes that need time rather than traffic.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Progressive delivery is the composition of three things: a rollout mechanism that can express partial exposure, a comparison that decides whether the current step is healthy, and a control that promotes, holds or aborts based on that comparison.
- The key structural idea is that there are two independent progressions. The artifact progresses through the fleet: 1% of traffic, then 5%, then everything. The feature progresses through the audience: internal users, then one tenant, then a percentage, then everyone. They are separate dials, they move at different speeds, and either one alone leaves a gap.
- The gate is what turns a ramp into delivery rather than a slower deploy. It compares candidate against a concurrent baseline on a defined set of signals, and its default action on ambiguity is to stop (Canary Analysis: Compared Against What?).
- Abort must be automatic and promotion may be automatic. That asymmetry is the design: a machine that stops on a bad signal is a safety device, while a machine that continues on a weak one is a faster way to reach everyone.
- Everything upstream is a precondition. Progressive exposure requires that partial exposure be *safe*, which means version coexistence, backward-compatible state and a reversal that works at every step (Version Coexistence: N and N+1, in Both Directions).
Two dials, not one
This is the idea that distinguishes progressive delivery from "canary plus flags". The artifact progresses through infrastructure; the feature progresses through people. A change usually needs both, and each is blind to the other's failures.
A dependency upgrade needs the artifact dial and cannot be flagged. A pricing change needs the audience dial and is unaffected by traffic weight. A rewrite of the checkout path needs both at once, and the interesting question is which order.
The loop, step by step
Every progressive rollout is the same five-stage loop repeated per step. Written out, it makes visible where implementations usually cut a corner: the observation window and the abort.
- 1Shift
Move the exposure dial to the next value — traffic weight, or targeting rule.
fails by The weight is applied to a version with no healthy instances, so the step carries no traffic at all.
evidence Requests observed on the candidate at approximately the intended share, across the endpoints that matter.
- 2Settle
Wait out warm-up: pools, caches, runtime. Comparison during this period measures newness, not the change.
fails by A short fixed pause that does not cover actual warm-up, so every step opens with a false regression.
evidence Candidate latency has stabilised relative to its own first minute.
- 3Observe
Hold the step for a window long enough to include the failure classes you care about.
fails by A window measured in minutes for a failure that needs a cache cycle or a scheduled job to appear.
evidence The window covered at least one full cycle of the slowest relevant process (A Successful Deploy Is Not Evidence of a Healthy System).
- 4Compare
Evaluate candidate against a concurrent baseline on errors, latency distribution, saturation and a business signal.
fails by Comparing against the fleet aggregate, or against yesterday, or on CPU alone (Canary Analysis: Compared Against What?).
evidence A recorded comparison with the query, the window and the result — not a screenshot of a dashboard.
- 5Decide
Promote to the next step, hold, or abort. Abort is automatic; promotion may be.
fails by Promotion by timeout when the comparison returned no data, which reads as healthy.
evidence The decision, its inputs and its actor are recorded on the same timeline as the deploy (The Audit Trail).
The two stages that are usually collapsed are Settle and Observe. Collapsing them produces a rollout that reliably detects crashes and nothing else.
Who decides to continue
The promotion decision is the one genuine design choice in a progressive rollout, and it is not a maturity ladder — the right answer differs by change and by how well the gate can see the failure that matters.
What advances the rollout from one step to the next?
when The failure signature is not expressible as a query — a visual regression, a change in what a response means, a first-of-its-kind change.
cost The rollout runs at the speed and attention of one person, and their attention degrades over a long ramp. It also does not scale past a few deploys a day.
when The default for significant changes. The machine catches what it can see quickly; the human decides to proceed.
cost Rollouts stall waiting for a human, which in practice means overnight rollouts do not progress.
when A well-understood change class with a comparison you trust, deploying often enough that human gating is the bottleneck.
cost The gate's definition of healthy becomes the definition of healthy, and failures outside it promote straight through to everyone.
when Failures in this system have historically been slow — leaks, drift, scheduled jobs — so time matters more than traffic share.
cost Very long rollouts that overlap with other changes, making attribution harder (Change Correlation).
when Genuinely trivial changes, or a system with too little traffic for a comparison to mean anything.
cost Full exposure immediately. Legitimate at small scale; the mistake is applying it by default to changes that are not trivial (Change Size: Why Small Changes Are Safer, and When They Are Not).
How to do it properly
Most important first.
- Build the ramp out of steps that multiply and windows that match the failure you are looking for, not out of a fixed timer.
- Automate the abort first and the promotion second. An automatic abort with a human promotion is a good system; the reverse is a fast way to ship a regression.
- Put artifact rollout and feature release on one timeline, so an operator can see both dials and both histories in one place (Deploys on the Same Timeline as the Symptom).
- Define the health comparison as a query against a concurrent baseline, on error rate, latency distribution, saturation and at least one business signal (Canary Analysis: Compared Against What?).
- Make the whole thing reversible at every step, and rehearse the reversal at each step rather than only at the last one (Rollback: Only Useful If It Is Actually Safe).
- Keep a manual override for both directions, with an audit trail — automation that cannot be overridden becomes something people work around (The Automation Trap).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by whichever dial is currently lowest — traffic weight or targeting rule — and by an automatic abort that ends the step without a human. It degrades to everyone when the gate cannot see the failure class, when the state change was applied fully at step one, or when the rollout controller itself fails open.
What can go wrong
- The gate is present but its thresholds are so loose that nothing ever fails it, so the ramp is ceremony that costs time and provides no evidence.
- The gate is so tight that it aborts on ordinary variation, teams learn to override it, and the override becomes the norm (Alert Fatigue in Observability terms is the same dynamic).
- Rollout automation with a bug of its own — promoting on a query that returns no data, which reads as "no errors".
- Long ramps overlapping each other, so at any moment several partially rolled-out changes are live and attributing a regression to one of them is guesswork (Change Correlation).
- Flags and rollouts owned by different systems, so reversing one leaves the other in place and the system ends up in a state that never existed in testing.
- Steps that expose the state change fully at step one, so the elaborate exposure control applies only to the part of the change that was never the risk (A Migration and a Deploy Are One Event).
- "Progressive delivery means we can ship anything safely." It bounds exposure to the code path. Schema changes, shared-state changes and external side effects are not on the dial (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- "The gate replaces judgement." The gate replaces *watching*. Whether a change should be behind a gate at all, and what to do when it aborts, is still judgement.
- "More steps is safer." More steps with windows too short to see anything is a longer rollout with the same evidence. The window matters more than the number of steps.
- "We have the tools, so we have the practice." Without an automatic abort that has actually fired, you have a slow deploy with a dashboard.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For each step: the candidate's signals compared against a concurrent baseline, the window that was observed, and the decision that was made — recorded, so it can be read after the fact.
- At least one rollout in recent memory that was aborted automatically, which is the only real proof the gate works.
- A single timeline showing artifact steps and flag changes together, which is what an incident responder actually needs (Reconstructing What Actually Happened).
- At any step, return the weight to the previous step or to zero. Exposure falls immediately for new requests; users already served are already served.
- If the feature is flagged as well, the flag is the faster reversal and should be the first move — it does not require the rollout system to be healthy.
- The state written during the ramp persists at every step, so the old version must remain able to read it for as long as it remains a rollback target (Version Coexistence: N and N+1, in Both Directions).
- A rollout system that is itself failing is a case worth planning for: know how to force full traffic to the last known good version without it.
- Automate: the ramp, the comparison, the abort, the annotation of the timeline, and the enforcement of the observation window.
- Automate promotion only for changes whose failure signature you are confident the comparison can detect. That is a per-change judgement, and it is a smaller set than it looks.
- Keep with humans: whether this specific change is a candidate for automatic promotion at all, and whether to override an abort (The Automation Trap).
- Rollouts get long. That is the cost, and it is paid on every change including the safe ones, unless you deliberately give small changes a shorter path.
- You now operate a rollout controller, a traffic-splitting layer, a flag service and a comparison query — four systems whose own failures can stop deployment entirely.
- Automated gates encode a definition of healthy. Anything outside that definition is invisible to the gate and easy to stop looking for (A Successful Deploy Is Not Evidence of a Healthy System).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICRollout controllers such as Argo Rollouts and Flagger implement the ramp-and-gate loop on Kubernetes; some managed platforms and serverless runtimes provide weighted traffic shifting with health gating as a product feature. Where none exists, the same loop can be run by a deployment pipeline that shifts weights and queries a metrics backend — slower and entirely workable. What must not vary is that the abort is automatic.
- SCALE-SPECIFICBelow the traffic level at which a small slice yields a readable comparison, the artifact dial does not work and only the audience dial does: release by cohort with flags, and verify by talking to those users. Building the full apparatus at that scale produces the ceremony without the evidence.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how much of the gate's definition of healthy should have been a test instead.