Canary Deployments
Send a small slice of real traffic to the new version, compare it against the old on the same signals, and only then commit the fleet.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How do I get evidence that a new version is good under real traffic before it serves all of it?
The last three incidents were caused by deploys that passed every test. We need to find out sooner, with a smaller blast radius, and without a manual staring contest.
Deploy one instance of the new version, watch the dashboard for a few minutes, and if nothing looks wrong, roll out the rest.
The canary carries 2% of traffic, so a defect affecting 1% of requests produces a handful of events — indistinguishable from noise on any dashboard.
- The canary carries 2% of traffic, so a defect affecting 1% of requests produces a handful of events — indistinguishable from noise on any dashboard.
- The dashboard shows aggregate metrics across both versions, so the canary's elevated error rate is diluted by the healthy majority and never becomes visible.
- The canary receives a biased slice: one large customer, or all traffic from one region, or only the requests that a hash happened to route there. It looks fine and represents nothing.
- "Watch for a few minutes" is unreliable in exactly the case that matters — the engineer watching is the one who wrote the change.
- The canary passes because the defect only appears once the new version handles the full write volume, and nothing about a 2% canary tests that.
What is actually happening
- A canary is a statistical argument, not a smoke test. You route a defined fraction of production traffic to the new version and compare its behaviour against the old version over the same interval, on the same signals, with the same traffic mix.
- It requires three capabilities that most services do not have by default: traffic splitting at a percentage, metrics labelled by version, and a decision rule that is written down before the deploy.
- The comparison must be against the concurrent baseline, not against yesterday. Traffic shape, dependency latency and load all change; the only fair control is the old version serving right now.
- Statistical power comes from events, not from time. A canary at 1% of a low-traffic service accumulates too few requests to detect anything, no matter how long it bakes.
- The canary shares the database, the cache and the queues with the stable version, so every compatibility rule from a rolling deploy still applies — a canary is a deliberately small two-version window (Rolling Deployments).
- Rollback is cheap because the canary is small: remove it from the routing pool, and the blast radius stops growing immediately.
Traffic split, and what the split is made of
The routing mechanism determines what the canary can tell you. Splitting by instance count means the canary share is tied to how many instances you run, and a small fleet cannot express 1%. Splitting per request at the proxy decouples the two and lets the share be a dial.
The second question is stickiness. A per-request split gives the cleanest statistics and can show one user two versions in consecutive requests. A session-sticky split is kinder to users and biases the sample toward whoever is active. Both are defensible; the failure is not choosing.
Decide the rule before you deploy
A canary that is evaluated by looking at graphs afterwards will be promoted, because the person looking wants it promoted and every graph has noise in it. The value comes from committing to the comparison in advance: these signals, this threshold, this window, this action.
Include at least one signal that a technically-successful-but-wrong version would move. Error rate and latency do not catch a handler that returns an empty list instead of results.
1# Illustrative shape, not a specific tool's schema.2canary:3 stages: [1, 5, 25, 100] # percent of traffic4 bakePerStage: 10m5 minRequestsPerStage: 5000 # power, not patience: below this, do not decide6 warmupIgnore: 2m # cold pool / JIT — do not judge the first minutes7 8 compareAgainst: baseline # the stable version, same window — never yesterday9 10 metrics:11 - name: error_rate12 query: rate(http_requests_total{status=~"5..",version="$v"}[5m])13 failIf: canary > baseline * 1.214 15 - name: latency_p9916 query: histogram_quantile(0.99, http_request_duration_seconds{version="$v"})17 failIf: canary > baseline * 1.318 19 - name: db_calls_per_request # catches an N+1 introduced by the change20 query: rate(db_queries_total{version="$v"}[5m]) / rate(http_requests_total{version="$v"}[5m])21 failIf: canary > baseline * 1.522 23 - name: checkout_success_rate # the business signal: 200s that did nothing24 query: rate(checkouts_completed_total{version="$v"}[5m]) / rate(checkout_attempts_total{version="$v"}[5m])25 failIf: canary < baseline * 0.9526 27 onFail: abort # weight -> 0 immediately, page the deployer28 onPass: advanceThree details do the work. minRequestsPerStage refuses to decide on insufficient evidence rather than deciding badly. warmupIgnore prevents a cold canary from failing on its own coldness. db_calls_per_request catches regressions that neither latency nor errors reveal at 1% traffic — the query count per request changes immediately, while its latency effect only appears at full load.
Canary, rolling or blue-green
The three strategies are not a maturity ladder. They answer different questions: rolling asks "can the versions coexist", blue-green asks "can I switch instantly and switch back", canary asks "is the new version measurably as good". A team can use all three for different services and be right every time.
The deciding inputs are traffic volume, the cost of spare capacity, and whether per-version telemetry exists. Adopting canary without the third is the most common wasted investment in this area.
What is the dominant risk in this deploy?
when Routine changes, compatible by construction, capacity is the constraint.
cost A long two-version window; rollback is another full rollout (Rolling Deployments).
when Low traffic, cannot canary meaningfully, want automatic abort.
cost Detects only failures loud enough to move fleet-wide signals.
when High traffic, behaviour risk, per-version metrics exist.
cost Traffic-splitting and analysis infrastructure; longer deploys.
when Instant rollback matters more than capacity cost; version coexistence is genuinely hard.
cost Double capacity during the switch, and the database is still shared (Blue-Green Deployments).
when The risk is in the behaviour, not the deployment.
cost Flag state becomes another version dimension to reason about (Feature Flags: Rollout, Kill Switches and Debt).
How to build it
Most important first.
- Label every metric, log and span with the version. Without per-version signals a canary is theatre — this is the prerequisite, not a nice-to-have (The Metrics a Backend Must Emit).
- Decide the signals and thresholds before deploying: error rate, latency percentiles, saturation, and one or two business metrics that would catch a semantically wrong but technically successful response.
- Prefer request-level random splitting to instance-count splitting when the platform supports it, and be explicit about whether a user is pinned to one version for their session — an unpinned canary can show a user two different behaviours in a row.
- Size the canary so it accumulates enough events to detect the effect you care about in the bake time you are willing to spend. If it cannot, do not run a canary — run a slower rolling deploy with a health gate instead.
- Include the signals a canary uniquely can catch: a new query pattern's effect on database load, a change in cache hit rate, a change in outbound call volume per request (The N+1 Query Problem).
- Automate the decision. An analysis step that compares canary and baseline and either promotes or aborts removes both the human bias and the 3am judgement call.
- Progress in stages — 1%, 5%, 25%, 100% — with a gate at each. Most defects show up at the first stage; saturation defects show up at the later ones.
What can go wrong
- The canary looks perfect and the full rollout fails, because the defect is load-dependent: a lock, a pool limit or a cache miss rate that only bites at full volume (Connection Pool Exhaustion).
- The canary instance is cold — empty caches, unwarmed JIT, a fresh connection pool — so its latency is worse for reasons unrelated to the change, and the deploy is aborted for the wrong reason (JIT and Warm-Up: The First Thousand Requests Are a Different Program in Observability & Performance).
- A canary that writes data the stable version cannot read. The blast radius of a *traffic* canary is small; the blast radius of the data it wrote is not.
- Automated analysis on a metric with too few samples, producing alerts on noise and training everyone to ignore it.
- Long-running canaries that become permanent, leaving two versions in production indefinitely and doubling the compatibility surface.
- Both versions serve concurrently, so every cross-version compatibility hazard from a rolling deploy is present, deliberately, for the whole bake time.
- An unpinned split can send a user's write to the new version and their immediate read to the old one, surfacing any behavioural difference as an apparent data inconsistency.
- Abort and promote can be triggered concurrently by an automated analyser and a human. The routing change must be a single serialised decision.
- Both versions enforce authorization independently. A canary containing a permissions change means a fraction of requests are evaluated under different rules — fine if intended, dangerous if unnoticed.
- Do not select canary traffic by a client-controllable header unless that header is validated. A version selector an attacker can set is a way to reach code that has not finished review (The Trust Boundary).
- Canary analysis dashboards often expose per-customer breakdowns. Treat them as production data with the same access controls.
- "A canary is a smoke test in production." A smoke test asks "does it respond". A canary asks "is it statistically indistinguishable from the version we trust".
- "The canary was healthy, so the release is safe." It is safe *at that traffic share*. Saturation, contention and cache-miss effects are not proportional.
- "Canary means one instance." Instance-count splitting is one implementation and a coarse one; the useful definition is a controlled fraction of traffic with a controlled comparison.
- "We canary, so we do not need expand-contract." The canary shares the schema with the stable version. It makes compatibility more important, not less.
Operating it
- Two panels side by side, canary and baseline, for the same signals over the same window. If you cannot produce that view, you cannot run a canary.
- Error rate, p50/p95/p99 latency, saturation of the pool and the loop, and outbound calls per request — the last one catches a whole class of regressions that latency alone hides.
- A business signal: checkout completions per thousand requests, or whatever the service exists to do. A version that returns 200 for everything and does nothing is invisible to technical signals.
- Record the canary decision — promoted or aborted, on what evidence — as an artifact, so a later incident can be traced to the deploy that was allowed through.
- Canaries need volume. Below a certain request rate they cannot detect anything in reasonable time, and a staged rolling deploy with a health gate is the honest alternative (Rolling Deployments).
- At high volume a 1% canary is thousands of requests per minute, which detects small regressions quickly and makes automated analysis genuinely reliable.
- With many services, canary infrastructure becomes a platform capability rather than a per-service script — and the analysis rules become a shared, reviewed asset.
- Canarying needs traffic splitting, per-version telemetry and an analysis step. That is real infrastructure, and it is wasted on a service that deploys twice a month.
- Bake time lengthens every deploy, which reduces deploy frequency, which tends to increase the size of each change — the opposite of what you wanted.
- A canary reduces the blast radius of serving, not of writing. Data written by a bad canary is as permanent as data written by a bad full deploy.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe idea — controlled fraction, concurrent baseline, pre-declared decision rule — is platform-independent.
- SCALE-SPECIFICRequires enough requests per unit time for a small fraction to be statistically meaningful. A service handling a few requests a minute cannot canary usefully; use a staged rollout with a health gate instead.
- CLOUD-SPECIFICHow traffic is split differs: some load balancers support weighted target groups, service meshes split per request with header-based overrides, and simple platforms only approximate a split by instance count — which couples canary share to capacity.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — progressive delivery, error budgets and the decision to spend one on a release.