Canarying a Model or Prompt Change
Roll a behaviour change to a small share of traffic and judge it on quality, cost, latency and safety — where quality is the one that is not a threshold.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you evaluate a model or prompt change on real traffic before it reaches everyone?
An eval suite scores a change against a dataset you assembled. Production sends inputs you did not think of, at a distribution you did not choose, from users who react to answers in ways no dataset captures. Something has to sit between the gate and full traffic.
The evals passed, so deploy it. If something is wrong, the error rate and latency dashboards will show it, exactly as they do for every other service.
A degraded agent usually has a normal error rate. It returns well-formed, confident, plausible answers that are worse — no exception, no non-200, nothing for an error-rate alert to fire on.
- A degraded agent usually has a normal error rate. It returns well-formed, confident, plausible answers that are worse — no exception, no non-200, nothing for an error-rate alert to fire on.
- Latency and cost can move in the direction that looks like an improvement while quality falls, because a model that gives up early is fast and cheap.
- By the time a quality regression is visible in business metrics — escalations, retries, thumbs-down, task completion — it has affected everyone for hours.
- Rolling back after full exposure does not undo the actions an agent already took through its tools, which is a class of damage that ordinary services do not have.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The mechanism is ordinary progressive delivery — a small traffic share, a comparison against a baseline, and expansion only if the comparison holds (Canary: One Percent, Then Five, Then Watch, Canary Analysis: Compared Against What?). What differs is entirely in what you compare.
- Four signals matter: quality, cost, latency and safety. Three of them are thresholds you can automate. Quality is not.
- Quality is hard for three compounding reasons: there is often no ground truth for a live request, the outputs are variable so a difference needs enough samples to be distinguishable from noise, and the label itself is a judgement — usually a model's or a human's, not a measurement.
- That makes an AI canary longer and larger than a deployment canary. An error-rate regression is visible in minutes at 1%; a quality regression may need hours of traffic before the difference separates from variance.
- Cost and latency, by contrast, become clearer signals here than in ordinary services, because they are per-request variables rather than fixed properties. A change that adds tool calls or lengthens context shows up immediately (Agent Cost in Production).
- Safety is a gate, not a metric to trade off. A single instance of a forbidden action is a stop, regardless of what the aggregate quality score did (Input and Output Guardrails).
- Cohort choice matters more than in a normal canary: routing 1% of *requests* mixes versions within one user's session, which produces incoherent behaviour and contaminates the comparison. Route by user or session instead.
The four signals, and which of them is a threshold
Automatable means the machine can decide without a person reading anything. Three of the four qualify. The fourth is why an AI canary needs a human at the expansion decision.
| Signal | What you compare | Is it a threshold? | How it fools you |
|---|---|---|---|
| Safety | Forbidden tool calls, guardrail trips, policy violations, refusal-rate spikes | Yes — and it is a stop, not a trade-off | Only counted when it fails loudly; a harmful action that succeeded looks like a normal tool call (Tool Misuse and Data Exfiltration) |
| Cost | Per-request cost distribution: tokens in and out, tool calls, loop iterations | Yes, once you set a ceiling | Averages hide a small share of runaway requests; watch the tail (What One Agent Run Costs, and Which Term Dominates) |
| Latency | End-to-end and per-step latency distributions, especially p95 and p99 | Yes | A model that answers with less work is faster and may be worse (Inside One Model Call: Queue, First Token, Generation) |
| Quality | Task completion, user retries, escalations, thumbs-down, plus graded samples of candidate versus baseline | No — no ground truth, high variance, and the label is a judgement | Small deltas within noise are read as results; aggregates hide segment regressions |
A rollout schedule that can actually say something
Each step has an entry condition and a stop condition, and the stop conditions are not symmetrical: safety stops immediately, cost and latency stop on a ceiling, and quality holds the rollout rather than reversing it while a human looks.
The duration column is the part people compress. Holding at a step is what buys the sample size, and cutting it short converts the whole exercise into theatre.
- 1Shadow (optional)
Runs the candidate on copied traffic without executing tool calls; compares intended actions and responses to the baseline.
fails by Shadow tool calls executed by accident, producing real side effects from a version serving nobody.
evidence A diff of intended actions, and confirmation that the shadow path cannot write.
- 2Internal cohort
Routes staff or opted-in users to the candidate for qualitative reading of real outputs.
fails by Internal traffic is unrepresentative, so it exercises a different distribution than customers do.
evidence Human-read samples with specific observations, not a thumbs-up.
- 31% by user
First real exposure. Watches safety and cost closely; quality begins accumulating samples.
fails by Routing per request instead of per user, splitting sessions across versions.
evidence Cohort assignment is stable per user, and the baseline is running concurrently.
- 4Hold and measure
Accumulates enough traffic for a quality comparison to separate from variance.
fails by Advancing on schedule rather than on sample size, so the pass means nothing.
evidence A stated sample size and the metric's variance, not just a delta.
- 5Expand stepwise
Increases share, re-checking all four signals and segment breakdowns at each step.
fails by Only aggregates checked, so a regressed segment is averaged away.
evidence Per-segment comparison at each step, for the segments that matter to the product.
- 6Full traffic
Completes the rollout, with online evaluation continuing on sampled requests.
fails by Measurement stops at 100%, so drift after rollout is invisible.
evidence A quality signal on the operational dashboard after the rollout ends.
- 7Retire the baseline
Removes the old version once the new one is established and the kill switch targets it instead.
fails by Baseline retired before the kill switch is repointed, leaving no fallback (The Agent Kill Switch).
evidence A tested fallback path that does not require a deploy.
The optional shadow step is worth the most for agents that take actions. Comparing intended actions is a far stronger safety signal than comparing text, and it costs no user exposure at all.
What to do when quality is ambiguous
The realistic outcome of an AI canary is not pass or fail. It is a small delta with meaningful variance, a couple of segments that moved in opposite directions, and a decision to make. These are the options, and none of them is free.
What is the right next move?
when The metric could reach significance in a tolerable window and the exposure risk is low
cost Users stay on an unproven version for longer, and the rollout occupies the team's attention
when Safety is clean and the downside of the change is a slightly worse answer, not a harmful action
cost More users exposed to something you have not yet shown to be better
when The proxies are weak and the change was made for a reason a human can evaluate directly
cost Slow and expensive, and human grading has its own variance and its own biases
when The change targeted cases your dataset barely covers, which is why the gate was uninformative
cost The work is deferred; the dataset improvement is the real deliverable (Golden Datasets)
when The change was made to reduce cost or latency and quality parity was the requirement, not improvement
cost Parity within noise is not parity; you are accepting a possible small regression deliberately, and should say so
when Prompt, model and tool changes were bundled, so the ambiguous result cannot be attributed
cost More rollouts and more elapsed time — but every subsequent canary becomes interpretable (Change Size: Why Small Changes Are Safer, and When They Are Not)
How to do it properly
Most important first.
- Start at a small share and route by user or session, so a given user gets one consistent version and their outcome metrics are attributable.
- Run the baseline concurrently. Comparing a canary today against production last week confounds the change with traffic and time-of-day effects.
- Gate safety and deterministic criteria automatically: forbidden tool calls, schema failures, refusal-rate spikes, guardrail trips. These stop the rollout on a single occurrence or a tight threshold.
- Watch cost and latency per request as distributions, not means — a change that makes a small share of requests loop is invisible in an average and obvious at the tail (Tail Latency: Why p50 Being Fine Does Not Help).
- For quality, combine cheap proxies available in real time — task completion, user retry, thumbs-down, escalation, conversation length — with an offline graded sample of canary versus baseline responses (LLM-as-Judge).
- Decide the expansion schedule and the stop conditions before starting, and write down how long you will hold at each step. Otherwise the canary lasts until someone is impatient.
- For a change with tool side effects, consider shadow evaluation first: run the candidate alongside production without executing its tool calls, and compare intended actions (Shadow Traffic: Real Requests, Discarded Answers).
- Keep the kill switch independent of the rollout mechanism, so stopping does not require a deploy (The Agent Kill Switch).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The routing share is the containment, and it is only real if the version is chosen per user rather than per request. For an agent with tool side effects, containment also depends on which of those actions can be compensated.
What can go wrong
- Requests routed randomly rather than by user, so one conversation is served by two versions and both the experience and the measurement are corrupted.
- A canary share so small that no quality difference could reach significance in any reasonable window, giving a confident pass that means nothing.
- The canary cohort being unrepresentative — internal users, one region, one tenant — so it exercises a different distribution than the traffic it is meant to predict.
- Watching only the aggregate, while a specific segment or task type regresses badly and is averaged away.
- A judge model used for canary grading that is itself changed mid-rollout, moving the comparison rather than the system.
- Automatic rollback wired to a noisy graded score, producing reversals of good changes and a team that stops trusting the gate.
- Cost regressions accepted silently because the quality improved, with no ceiling — a rollout that succeeds on quality and multiplies spend is not a success (Budgets, Deadlines and Step Limits).
- "Error rate and latency are fine, so the canary is healthy." Those are the two signals a quality regression is most likely to leave untouched.
- "Quality is up 2%, ship it." Without knowing the metric's variance and the sample size, 2% may be indistinguishable from nothing.
- "Canary at 1% for ten minutes." That is a deployment canary schedule. A behaviour change usually needs more traffic and more time to say anything about quality.
- "The canary passed, so the change is safe for everyone." It is safe for the cohort you routed, on the traffic you saw. Segment-level checks and continued online evaluation are still required (Regression Gates and Online Evaluation).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The canary report names all four signals with the baseline value beside each, not just error rate and latency.
- The quality comparison states its sample size and how variable the metric is, so a small delta is not read as a result.
- Cost and latency are reported as distributions with tail values.
- A stopped rollout exists in recent history, with the signal that stopped it.
- Route traffic back to the baseline version. Because a canary is a routing decision, reversal is fast and does not need a rebuild (Progressive Delivery: Exposure as a Dial).
- Actions already taken through tools are not reversed by routing. For agents with side effects, know before the canary which actions are compensable and which are not (Approval Gates and Risk Classes).
- If the canary is a model change and the identifier was a moving alias, reverting your configuration may not revert the provider's behaviour (Prompts and Models Are Deployables).
- Automate the routing, the concurrent baseline, the metric collection and the safety stop.
- Automate rollback on safety trips and on hard cost or latency ceilings, where the criterion is unambiguous.
- Do not automate expansion on graded quality alone. Automate the report; keep the decision to expand human until you have calibrated the metric against human judgement and know its variance (Eval Metrics: What to Measure and How).
- Automate the collection of canary disagreements — cases where candidate and baseline differ materially — into the eval dataset, since those are the highest-value cases you will find (Golden Datasets).
- A canary long enough to detect a quality difference exposes real users to a possibly worse experience for hours, which is a deliberate cost.
- Running a concurrent baseline means paying for both versions during the rollout, and for graded evaluation of both.
- Small shares protect more users and detect less. That tension has no clean answer; it is a decision about how much of a regression you are willing to miss.
- Cohort routing by user gives coherent experiences and a less random sample, which is usually the right trade but does bias the comparison if the cohort is not chosen carefully.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALProgressive exposure with a concurrent baseline applies to any behaviour change on any provider. The share, the duration and the metrics are all workload-dependent and none of them transfer between systems.
- SCALE-SPECIFICBelow a certain traffic volume a quality canary cannot reach useful confidence in any acceptable window. At that scale, use a larger share with a human reviewing a sample of real outputs, and accept that the evidence is qualitative.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.