SafetyGENERALSCALE-SPECIFIC
Canary Analysis: Compared Against What?
Deciding whether a candidate is healthy by comparing it against a concurrent baseline on errors, latency, saturation and business outcome — and never on CPU alone.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
The canary is running. What do I compare, what do I compare it to, and what result means "continue"?
The problem
A canary that nobody can read is a slower deploy with the same risk. The decision at the end of every step is a comparison, and the quality of the rollout is entirely the quality of that comparison.
What teams do first
Look at the canary's CPU and error count. If CPU is similar to normal and there are no obvious errors, promote.
How it breaks
CPU alone decides nothing. A candidate can be slower, wrong, dropping requests, exhausting a connection pool or returning empty results at exactly the CPU of the old version — and a change that computes *less* because it fails early looks like an improvement.
How it breaks in production
- CPU alone decides nothing. A candidate can be slower, wrong, dropping requests, exhausting a connection pool or returning empty results at exactly the CPU of the old version — and a change that computes *less* because it fails early looks like an improvement.
- "Similar to normal" compares against a remembered number. Normal varies by hour, by day and by traffic mix, so a comparison against memory is a comparison against the wrong baseline.
- An error *count* on one percent of traffic is one percent of the errors, so it looks small no matter how bad the candidate is. Only the rate is comparable.
- A fresh process is cold: empty caches, unestablished pools, unwarmed runtime. Its first minutes look worse for reasons that have nothing to do with the change, and teams learn to discount the beginning of every window — including the times when the beginning was the signal.
- Some failures are not visible in service-level signals at all. The service returns 200 with an empty list; the checkout succeeds and charges nothing; the job runs and processes zero rows.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A canary analysis is a comparison of two populations observed at the same time: the candidate, and a baseline running the previous version under the same traffic and the same conditions.
- Concurrency is what makes the comparison valid. It controls for time of day, traffic mix, dependency behaviour, neighbouring load and every other thing that changed between yesterday and now — none of which you can control for by remembering.
- The baseline must exclude the candidate. Comparing the candidate against the fleet aggregate compares it against a number that includes it and is dominated by the old version, which dilutes exactly the signal you are looking for.
- Four families of signal, and you need more than one: errors (rate, by class, not count), latency (a distribution — the tail moves before the median does), saturation (the resource that is closest to its limit, which is rarely CPU), and business outcome (did the thing the system exists to do still happen).
- A difference must be large enough to be distinguishable from ordinary variation over the window you observed. A one-instance canary over two minutes produces a small sample, and small samples produce differences that mean nothing in both directions (Regression or Tuesday? Telling a Real Change from Noise).
- The output is a decision — promote, hold, abort — and the default on an unreadable comparison must be to stop.
Four families of signal, and what each one alone would miss
GENERALThe families are universal; which specific saturation signal matters is entirely system-specific — a database-bound service saturates on connections, a queue consumer on lag, a memory-heavy service on allocation rate. Naming the one that constrains *your* service is part of the analysis design, and it is not usually CPU.
The reason to insist on more than one family is not thoroughness, it is that each family has a specific blind spot and the blind spots do not overlap. The row to read twice is the third: do not decide from CPU alone, because the most common way a broken candidate hides is by doing less work.
| Signal family | What to compare | What it catches | What it misses entirely |
|---|---|---|---|
| Errors | Rate by class — 5xx, 4xx, exceptions, timeouts — candidate versus concurrent baseline | Crashes, unhandled paths, dependency failures, bad deserialization | Anything that returns a successful response containing the wrong thing |
| Latency | The distribution, at the percentiles users feel — not the mean | Regressions that show in the tail long before the median moves (Tail Latency: Why p50 Being Fine Does Not Help) | Failures that are fast: a candidate erroring immediately is quicker than one working |
| Saturation | The resource nearest its limit: pool, queue depth, memory, descriptors — and yes, CPU among them | Leaks, exhaustion, throttling, capacity regressions (USE: Utilization, Saturation, Errors) | Correctness of any kind. CPU alone is the classic false pass: a candidate that fails early consumes less of it |
| Business outcome | The thing the system exists to do: orders, deliveries, jobs completed, messages sent | The whole class where every technical signal is green and the product is broken | Anything slower than the observation window; anything with a cause outside the system |
Three baselines, only one of which is valid
Nearly every unreliable canary analysis fails here rather than in the choice of signals. The comparison is against the wrong population, and no amount of statistical care fixes a bad control.
What "compared to normal" turns out to mean
Against the fleet, or against yesterday
candidate error rate 0.9%
vs "fleet error rate" 0.1%
-> but the fleet number INCLUDES the candidate
and is 99% old version, so it barely moves
candidate p99 320ms
vs "p99 last Tuesday" 300ms
-> different traffic mix, different hour,
a dependency was upgraded in betweenAgainst a concurrent baseline
baseline: same version as before,
same instance size, same zone,
serving now, excluded from the candidate slice
compare candidate vs baseline
over the same window, same traffic mix,
same dependency conditions, same hour
-> the only difference left is the changeA control group has to differ from the treatment in exactly one respect. The fleet aggregate contains the candidate and is dominated by the baseline version, so it dilutes the signal; a historical window differs in traffic, load and dependency behaviour as well as in version. Only a concurrent, separate baseline isolates the change.
How a canary analysis passes something it should have caught
Every row here ends with a green rollout and a subsequent incident. They are worth reading as a checklist against your own gate, because most of them are configuration rather than concept.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Decision made on CPU and a glance at the error graph | Candidate promoted; error rate climbs at full traffic | The candidate failed fast on a subset of requests, using less CPU and less time — both signals moved in the reassuring direction | Require error rate by class and a business signal; never gate on a resource metric alone |
| Comparison against fleet aggregate | Nothing detectable at any step | The candidate is a small share of a number dominated by the version it is being compared to | Query candidate and baseline as separate series by version label |
| Metrics pipeline broken for the new version | Gate reports healthy and promotes automatically | An empty result set evaluated as "no errors" | Treat missing data as a failed check; assert minimum request volume before evaluating anything |
| Two-minute observation window | Candidate passes; memory exhaustion appears hours later | Leaks, connection growth and cache-driven failures need elapsed time, not traffic share | Hold the final pre-full step for longer than one cycle of the slowest relevant process (A Successful Deploy Is Not Evidence of a Healthy System) |
| Warm-up not excluded | Every canary shows an early latency spike; team stops looking at the first minutes | Cold caches and unestablished pools, misread as a regression | Measure warm-up duration, exclude it explicitly, and keep looking at what happens after it |
| Business metric too slow to move | Gate has a business signal that never varies within a step | A daily aggregate cannot resolve a ten-minute window | Choose a business signal with the right time resolution — completed checkouts per minute, not revenue per day |
| Candidate slice excludes expensive endpoints | Green throughout; failure at full traffic on one route | The routing key correlated with the traffic type | Verify the candidate's endpoint mix matches the baseline's before trusting the comparison |
How to do it properly
Most important first.
- Compare against a concurrent baseline of the previous version, sized and placed like the candidate. If you cannot isolate a baseline, you do not have a canary analysis, you have a dashboard.
- Label telemetry by version so candidate and baseline are separable in every query (Deploys on the Same Timeline as the Symptom).
- Use rates and distributions, never counts and means. Error rate per class; latency at the percentiles that matter to users (Percentiles: Which One, and How Many Users Is That?, Tail Latency: Why p50 Being Fine Does Not Help).
- Include at least one business signal per service — orders placed, messages delivered, jobs completed — because it catches the class of failure where every technical signal is green (RED: Rate, Errors, Duration for the service-level trio, and this on top).
- Look at saturation of the constrained resource, whatever it actually is: connection pool, thread pool, queue depth, memory, file descriptors. It is usually not CPU (USE: Utilization, Saturation, Errors, Saturation: The Reading Utilization Cannot Give You).
- Exclude a defined warm-up interval, and know how long it is by measuring rather than guessing.
- Write the abort condition before the rollout starts, and make abort the action when the comparison is ambiguous or returns no data.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongOne percent
One testEveryone
What contains it
Contained by the exposure of the step being analysed — a failed analysis at one percent means one percent of requests were affected. The containment fails in one specific way: a comparison too weak to detect the failure lets the ramp continue, so a bad analysis converts a one-percent blast radius into an everyone blast radius one step at a time.
What can go wrong
Failure modes, including of the mitigation
- Comparison against the fleet aggregate, which includes the baseline version and buries the candidate's signal.
- Comparison against a historical window — yesterday, last week — so any difference in traffic or dependency behaviour reads as a change in the candidate.
- A query that returns no data being treated as "no errors", so a canary with broken telemetry promotes automatically.
- Thresholds set so loosely that nothing fails, giving the rollout the appearance of a gate without the function of one.
- Thresholds set so tightly that ordinary variation aborts rollouts, teams learn to override, and override becomes routine (Alert Fatigue in Observability terms).
- Warm-up counted as regression, or the warm-up exclusion set so long that it hides real early failures.
- A single business metric that is too slow to move within the window — measuring daily revenue on a ten-minute step tells you nothing.
Misreads this invites
- "CPU looked fine, so the candidate is fine." CPU is one saturation signal among several and often the least relevant. A candidate that fails fast uses less CPU, which looks like an improvement and is not.
- "No alerts fired, so the canary passed." Alerts are tuned for fleet-wide symptoms. A problem confined to one percent of traffic is usually below every alert threshold you have (Alert on Symptoms, Not on Causes).
- "The averages matched." The mean hides the tail, and the tail is where the users who notice live (Latency Is a Distribution, Not a Number).
- "Statistically significant means important." Given enough traffic, tiny differences become significant. The question is whether the difference matters to a user, which is a judgement, not a p-value (Benchmark Fallacies: Confident Numbers That Are Wrong).
- "The canary is green, so we are done." It is evidence about the sampled window at the sampled exposure. Verification continues after full rollout (A Successful Deploy Is Not Evidence of a Healthy System).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A recorded comparison per step: the queries, the window, the candidate and baseline values, and the decision. Not a screenshot.
- The candidate served a meaningful volume across the endpoints that matter, verified rather than assumed.
- At least one occasion where the analysis aborted a rollout that a human would have promoted — the only real proof that the comparison has power.
How you get back
- The output of a failed analysis is an abort: return the exposure to the previous step or to zero, which is a routing change rather than a redeploy (Canary: One Percent, Then Five, Then Watch).
- Aborting on an ambiguous signal is cheap and should be the default. Re-running a rollout costs a rollout; promoting a bad candidate costs an incident.
- The analysis does not reverse anything the candidate wrote, and it does not reverse external side effects it emitted during the step (Roll Forward: When Going Back Is the Harder Option).
What to automate, and what stays human
- Automate the comparison itself. A defined query evaluated the same way every time is more reliable than a person reading a dashboard while doing something else.
- Automate the abort, always. Automate the promotion only where you trust the comparison to see this change's failure mode.
- Do not automate the interpretation of a business-metric change. A drop in conversions might be the candidate or might be a marketing campaign ending; that judgement needs context the query does not have.
What this costs
- A concurrent baseline means running the old version deliberately alongside the new one for the whole ramp, which costs capacity and lengthens the rollout.
- Statistical rigour needs volume and time. Being confident about a small difference means a long window, and long windows overlap with other changes (Change Correlation).
- Every signal added to the gate is another way for a rollout to stall on something unrelated, so a gate with many signals needs tuning attention it will not always get.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe comparison logic — concurrent baseline, rates not counts, distributions not means, more than one signal family — is independent of platform and of metrics backend.
- SCALE-SPECIFICBelow the traffic volume where a small slice yields a distinguishable difference within a reasonable window, canary analysis cannot work and pretending otherwise produces confident wrong answers. At that scale, exposure control by cohort with direct feedback from those users is the honest substitute.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Cloudcanary
Observability & Performanceperformance-regression-detectionpercentilestail-latencysaturationuse-methodred-methodbenchmark-fallacies