Verify in Production
A deploy reporting success says the orchestration worked; verification is comparing the new version against a baseline on signals that reflect users.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you know a change is working, as opposed to knowing it was deployed?
Deployment tools report on their own work. They know whether instances started and passed health checks. They do not know whether the change did what it was supposed to do, and treating their green as the answer is how bad releases stay live until a customer complains.
The deploy went green and the dashboard looks normal. The change is out. Move on to the next thing.
Health checks answer "is the process alive", which almost every broken deploy also answers yes to. A service returning 500s for one endpoint is healthy by that definition (Probes: Readiness, Liveness and Startup).
- Health checks answer "is the process alive", which almost every broken deploy also answers yes to. A service returning 500s for one endpoint is healthy by that definition (Probes: Readiness, Liveness and Startup).
- "The dashboard looks normal" over an aggregate hides a change affecting a subset — one tenant, one region, one client version — inside a much larger denominator.
- Comparing against zero rather than against the previous version means you cannot tell an elevated error rate from your normal error rate.
- Some effects are delayed by design: a cache fills over an hour, a scheduled job runs at midnight, a queue backs up gradually. Verification that ends when the deploy ends misses all of them.
- Nobody defined what "working" would look like, so verification becomes an unstructured look at a dashboard, and its outcome depends on who is looking.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Verification is a comparison, and it needs three things: a signal that reflects users, a baseline to compare against, and a window long enough for the effect to appear.
- The baseline is the previous version serving comparable traffic. Comparing the canary against the stable version at the same moment controls for time-of-day, traffic mix and dependency weather in a way that comparing against yesterday does not (Canary Analysis: Compared Against What?).
- The signals that reflect users are the symptom-level ones: error rate, latency at the tail, and whatever business event the change was supposed to affect. Resource metrics are diagnosis, not verification (Alert on Symptoms, Not on Causes).
- Deploy annotations make the comparison possible after the fact: a marked timeline lets anyone see whether a shift lines up with a change (Deploys on the Same Timeline as the Symptom).
- The verification window is set by the slowest mechanism the change can affect. A change to a caching layer needs a window longer than the cache TTL to say anything.
- Verification must also cover the thing the change was for. A release that is technically healthy and did not produce the intended effect is a failed release, and only a business signal will say so.
Four layers of "it worked", from weakest to strongest
These are usually treated as one thing. They are not: each answers a different question, and only the last two say anything about users.
- 1Orchestration succeeded
The new version was placed and the old one removed.
fails by Reported as success while the process crash-loops just outside the observation window.
evidence Desired replica count reached and stable, not merely reached (Reconciliation: The Loop Under Everything).
- 2Process is healthy
The instance starts and answers its health endpoint.
fails by A health check that only proves the HTTP server is listening (Probes: Readiness, Liveness and Startup).
evidence Readiness reflects dependency availability, not just process liveness.
- 3Traffic behaves
Real requests on the new version succeed at the expected rate and latency.
fails by Aggregate metrics hide a subset failing; no per-version split.
evidence Error rate and tail latency for the new version compared against the old, at the same time (Canary Analysis: Compared Against What?).
- 4The change did its job
The effect the change was for actually happened.
fails by Nobody stated the expected effect, so nobody checks it.
evidence The named business or system signal moved as predicted, in the stated window.
The first two are about your infrastructure. Only the last two are about your users, and they are the two that require someone to have decided in advance what to look at.
A rollout that was verified, minute by minute
What verification looks like when it is doing its job. The change is behind a flag, rolled to a small cohort first, and the signal that stops it is a comparison rather than a threshold.
The important detail is at the eleven-minute mark: the aggregate error rate never left its normal band. Only the per-version comparison showed the problem, because the canary was a small share of traffic.
- T+0changeRelease 2026-08-26.3 deployed to 5% of instances; deploy annotation written
- T+1signalCanary instances pass readiness; traffic begins arriving on the new version
- T+3actionAutomated comparison starts: canary versus stable, error rate and p99, same window
- T+6signalCanary error rate 0.9% against stable 0.2%; still inside the configured tolerance
- T+9signalCanary p99 rising; stable p99 flat. Aggregate dashboard shows nothing unusual
- T+11actionComparison breaches: canary error rate 4.1% against stable 0.2%. Rollout halted automatically
- T+12signalOn-call paged with the comparison attached, not just a threshold breach
- T+13recoveryFlag for the new read path disabled; canary instances return to old behaviour
- T+15recoveryCanary error rate returns to 0.2%; user impact ended at roughly 5% of traffic for 12 minutes
- T+40actionCause identified in a code path only reached for one tenant's data shape; fix planned as a slice
The flag flip, not the rollback, ended the impact — it was available in seconds where a redeploy would have taken minutes. The rollback happened afterwards, unhurried.
Verification failures
The characteristic failure of this stage is not a wrong answer. It is a confident answer to a question nobody meant to ask.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Deploy tool reports success | Bad version serves for hours until a customer reports it | Orchestration success read as behavioural success | Add a post-deploy comparison as a required step, not a habit (A Successful Deploy Is Not Evidence of a Healthy System) |
| Aggregate dashboard checked | One tenant fully broken, error rate barely moves | Large denominator hides a small absolute number of failures | Slice by version and tenant; alert on the slice (Blast Radius: If This Is Wrong, How Much Does It Affect?) |
| Canary green in two minutes | Failure appears an hour later | Window shorter than the cache TTL or the job schedule the change affects | Set the window from the slowest affected mechanism |
| Health check passes | Requests fail while instances report healthy | Health endpoint tests the web server, not the dependency the change touched (Probes: Readiness, Liveness and Startup) | Readiness should reflect ability to serve, including critical dependencies |
| Canary threshold never fires | Confidence in a gate that has never stopped anything | Thresholds set to avoid false positives, and never tested against a real bad release | Exercise the gate deliberately; a gate that has never fired is untested code |
| Change deployed with no stated expected effect | Release is healthy but achieved nothing, discovered weeks later | Verification defined as absence of harm only | State the intended effect at plan time and check it at verify time (Plan and Code) |
How to do it properly
Most important first.
- Write down, before deploying, what signal would show this working and what value counts as normal. Verification without a pre-stated expectation drifts into rationalisation.
- Compare the new version against the currently serving version, on the same signals, at the same time (Canary Analysis: Compared Against What?).
- Slice the signals by version, and by tenant or region where the change could affect them unevenly. Aggregates hide exactly the failures progressive rollout exists to catch (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Annotate deploys on the dashboards operators actually use, so the correlation is visible without anyone digging (Change Correlation).
- Keep the rollout paused long enough to cover the slowest affected mechanism, not long enough to feel thorough.
- Verify the intended effect, not only the absence of harm — the conversion, the queue draining, the job completing (A Successful Deploy Is Not Evidence of a Healthy System).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The canary cohort, if there is one, and the rollout being halted when the comparison fails. Without progressive rollout the scope is everyone, and verification only shortens the time to detection.
What can go wrong
- Automated canary analysis with thresholds so loose it never fails, which is worse than no analysis because it is trusted.
- Thresholds so tight that they fail on noise, so the team learns to override them — the alert-fatigue failure applied to rollouts (Alert Fatigue).
- Canary receiving unrepresentative traffic: a new instance with a cold cache, or traffic routed by a hash that puts one large tenant on it every time.
- Verification run only on the first cohort, then the rest of the rollout proceeds unwatched.
- Signals with too much delay to be useful for a rollout decision — a daily batch metric cannot verify a deploy.
- "The deploy succeeded" means the change works. It means the orchestrator finished its work (A Successful Deploy Is Not Evidence of a Healthy System).
- "No alerts fired" means it is fine. It means nothing crossed a threshold someone set in advance for a failure mode they anticipated.
- "Canary is green after two minutes" — two minutes verifies startup, not behaviour. Anything cache-, queue- or schedule-dependent has not happened yet.
- "We monitor everything, so we do not need a verification step." Monitoring is standing capability; verification is an actual comparison someone performs on a specific change.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A recorded comparison: candidate versus baseline, on named signals, over a stated window.
- The rollout was actually stopped at some point in the past by that comparison. A gate that has never fired is untested.
- Deploy markers appear on the dashboards the on-call engineer opens first.
- The business signal the change targeted moved in the expected direction, and someone checked.
- Verification is what triggers rollback, so its own failure is silent: nothing happens, and a bad version keeps serving.
- If the comparison fails, stop the rollout first and decide afterwards. Halting is cheap and reversible; continuing while you investigate is not (Rollback: Only Useful If It Is Actually Safe).
- Roll back the cohort, not the whole system, where the strategy allows it — that is what progressive delivery bought you (Progressive Delivery: Exposure as a Dial).
- If the change is behind a flag, the flag is the faster lever: a config flip beats a redeploy every time (Feature Flags: Deploy Is Not Release).
- Automate the comparison: pull candidate and baseline signals, apply the stated criteria, halt the rollout automatically on breach.
- Automate deploy annotation, because a manual annotation is the first thing dropped during a busy week.
- Keep human: the roll-back-or-forward decision when the comparison is ambiguous, and any override of a failing gate — which should require a reason that is recorded (The Automation Trap).
- A real verification window slows every deploy, including the overwhelming majority that are fine. That is a direct tax on delivery speed.
- Automated canary analysis needs enough traffic on the canary for the comparison to mean anything; on a low-traffic service the statistics do not support the decision and a longer window or a different strategy is honest (Canary: One Percent, Then Five, Then Watch).
- Per-version, per-tenant signal slicing multiplies metric cardinality, which has a real cost in the observability system (Cardinality: The Label That Took Down Monitoring).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALComparison against a baseline on user-visible signals applies to any deployment mechanism. What changes is the granularity available: a platform without traffic splitting can compare before and after in time, which is weaker because it does not control for what else changed.
- SCALE-SPECIFICStatistical canary comparison needs enough requests in the window to distinguish a real change from noise. Below that — a low-traffic internal service — a longer soak, synthetic checks, or explicit manual verification are honest substitutes; pretending a five-request canary is evidence is not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — service level objectives as the pre-stated definition of "normal" that verification compares against.