Production Readiness Review
The conversation before a service carries real traffic: who owns it, how you will know it is broken, and how you get back.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does a service owe before it is allowed to carry production traffic, and who decides it is ready?
Services reach production because a deadline arrived, not because anyone checked whether they could be operated. The gaps — no owner, no dashboard, no rollback, no runbook — are discovered during the first incident, at the worst possible price.
It passes its tests, it deploys, and it works in staging. Ship it and we will add monitoring if we need it.
The first alert has nowhere to route. Several minutes of the first incident go to establishing whose service this is (The Ownership Record).
- The first alert has nowhere to route. Several minutes of the first incident go to establishing whose service this is (The Ownership Record).
- Nobody knows what healthy looks like, so nobody can tell that it has stopped being healthy. The failure is reported by a user.
- The rollback path is discovered during the incident. It usually turns out to have a step that needs a person who is asleep.
- Capacity was never estimated, so the first traffic spike or the first failover finds the limit empirically (Capacity During Failover).
- Dependency timeouts were never set, so a slow dependency becomes an outage rather than a degradation (Timeouts).
- "We will add monitoring if we need it" is only ever executed after you needed it. The work is real and unscheduled, so it competes with feature work and loses.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A readiness review is not a quality gate on the code. It is a check that the operational surface exists: the things an operator will need at 3am, when the author is unavailable and the person on call has never seen this service before.
- Every item on the checklist maps to a specific minute of a future incident. An owner saves the first five. A dashboard saves the next ten. A runbook saves the twenty after that. A rollback path decides whether the incident is minutes or hours.
- The checklist is a memory aid for a conversation, not a form. The value is in the discussion each item provokes — "what happens if the database is slow?" is worth more than a ticked box next to "timeouts configured".
- It works best as a design-time activity repeated at launch. Discovered at launch, a missing rollback story is an architectural problem with a week of work behind it and a deadline in front of it.
- The output is a status per dimension with evidence, deliberately not a score (The Readiness Scorecard). What makes a dimension READY is an artifact you can open, not an assertion in a meeting.
The thirteen dimensions
This is the checklist. Read the third column as the point of the exercise: each dimension exists because its absence costs a specific, predictable amount during a specific, predictable future minute.
The evidence column is what separates a real review from a form. "Yes" is not evidence. A link to a dashboard, a record of a rollback that happened, a restore that completed — those are.
| Dimension | The question | What its absence costs | Evidence that satisfies it |
|---|---|---|---|
| Ownership | Which team is paged, and who escalates? | The first minutes of every incident go to routing | A current record naming a team that exists (The Ownership Record) |
| Runbook | What does an operator do when the common failures happen? | Every incident is improvised from scratch | Entries with symptom, checks, mitigation and verification (Runbooks) |
| Dashboards | Can you see whether it is healthy in under a minute? | Diagnosis starts with building the view | A dashboard an operator who did not build it can read (Dashboards an Operator Can Act On) |
| Alerts | What pages a human, and does it correspond to user impact? | Customers detect your outages | Alert rules on user-visible symptoms, routed to the owning rotation (Alert on Symptoms, Not on Causes) |
| Capacity | What saturates first, and how much headroom is there? | The limit is discovered by traffic | A stated bottleneck and a headroom figure with its basis (Headroom) |
| Timeouts | What happens when a dependency is slow rather than down? | A slow dependency becomes your outage | A configured timeout on every outbound call (Timeouts) |
| Retries | What is retried, with what backoff and what budget? | Retries amplify a partial failure into a total one | Bounded retries with jitter, and an idempotency story (Without Jitter, Every Client That Failed Together Retries Together) |
| Backups | What data can be lost, and has a restore been done? | You have backups and no proven recovery | A completed restore with a measured duration (Restore Drills) |
| Rollback | How do you get back, and how long does it take? | Incidents last as long as a fix takes to write | A rollback performed at least once, deliberately (Rollback: Only Useful If It Is Actually Safe) |
| Health checks | How does the platform know an instance can serve? | Traffic reaches instances that cannot serve it | Readiness distinct from liveness, neither depending on the world (Probes: Readiness, Liveness and Startup) |
| Security | What identity does it run as, and what can that identity reach? | A compromise or a bug has unbounded reach | A scoped workload identity with no unused permissions (Workload Identity) |
| Dependencies | What does it need, and what happens when each is unavailable? | Failure modes are discovered one at a time in production | A dependency list with a stated degradation behaviour for each |
| SLO | What does "working well enough" mean numerically? | Reliability arguments have no shared reference | An objective with a measurable indicator behind it (SLOs: A Target, a Window, and a Reason) |
When to run it, and what it costs to run it late
The same review produces different value depending on when it happens. At design time, a missing rollback story is a paragraph in a document. At launch minus one week, it is a schedule problem. After launch, it is an incident with a postmortem attached.
The failure mode this ordering is designed to avoid is the one where the review discovers something architectural at the point where the only available responses are "launch anyway" or "slip the date". Both are bad, and the choice is usually made by whoever is under the most pressure.
- 1Design
Asks the questions whose answers shape the architecture: rollback, migration, data loss tolerance, failure behaviour of dependencies.
fails by Skipped because the service "is not real yet".
evidence A design document that states the recovery story, not only the happy path.
- 2Pre-launch
The full checklist against a service that exists but carries no traffic.
fails by Run as a form the week of launch, when nothing found can be acted on.
evidence Status per dimension with links, and dated owners for the gaps (The Readiness Scorecard).
- 3First traffic
Verifies the operational surface against reality — alerts fire, dashboards populate, the runbook matches what is actually deployed.
fails by Assumed to work because it was configured.
evidence An alert deliberately triggered and observed to route to a human.
- 4On material change
Re-assesses when the service becomes customer-facing, gains state, or takes on a new dependency.
fails by Never repeated, so the assessment describes a service that no longer exists.
evidence A dated re-review after the change that made the previous one stale.
The single highest-value item at the "first traffic" step is deliberately triggering an alert and watching where it goes. Alert routing is configuration that is never exercised until it matters, and it is wrong surprisingly often (An Alert Should Demand Action).
How much review does this service actually owe?
A uniform standard is either too heavy for the internal dashboard or too light for the payment path. Tiering is how the process stays credible — but tiering by importance is a conversation, not a formula, and the tier should be argued rather than assigned.
A new service is about to carry traffic. How much readiness evidence should it owe?
when Customer-facing, handles money or personal data, or is a dependency of other teams' services.
cost Days of engineering time and a scheduling dependency on a reviewer; will be resented if the reviewer lacks context.
when Internal service with a clear owner and limited blast radius.
cost Relies on the team being honest about gaps, which is a function of whether admitting a gap is safe.
when Low-risk internal tooling: who is paged, how do you know it is broken, how do you get back.
cost Misses capacity and dependency issues that will surface later — acceptable when the blast radius is genuinely small.
when Nothing carrying production traffic. A prototype behind a flag, with a named expiry.
cost Prototypes become load-bearing without anyone deciding to promote them; without an expiry this option is how unreviewed services enter production.
How to do it properly
Most important first.
- Run it twice: a lightweight version at design, when answers are still cheap to change, and the full version before production traffic.
- Require evidence rather than assertion. "Yes we have a runbook" becomes a link; "we can roll back" becomes the record of a rollback that was actually performed in a lower environment.
- Make the checklist a living artifact fed by your own incidents — a contributing factor that appears repeatedly becomes a checklist item (Learning Across Incidents).
- Keep it proportional. An internal tool with ten users and a customer-facing payment path do not owe the same evidence; say which tier applies and why.
- Let it block launch only for the small set of items where the answer is genuinely load-bearing: an owner, a way to detect failure, and a way to get back. Everything else can be a dated commitment.
- Re-run it when the service changes materially — it becomes customer-facing, it gains a datastore, it starts handling money, it gets a dependency on something it did not have before.
- Automate the parts that are machine-checkable and spend the human time on the parts that are not (Policy as Code).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Usually contained to the unprepared service — unless it becomes a dependency of others, at which point its missing operational surface is theirs too.
What can go wrong
- The review becomes a gate with a queue, so teams schedule around it and launch services that never went through it.
- Box-ticking: every item marked yes, no evidence attached, no conversation held. The document exists and the service is not ready.
- The checklist grows without pruning until it is sixty items, at which point it is completed by pattern-matching rather than by thinking.
- Applied only to new services, so the ones that predate the process — usually the oldest and most load-bearing — are never assessed.
- Owned by a central team with no context, producing generic findings that the service team correctly ignores.
- Passing the review treated as proof of reliability. It is proof that the operational surface exists, which is a much smaller claim (Parity That Is Worth Paying For).
- "Passing means the service is reliable." It means it is operable. Reliability is a property of running it, measured over time (SLOs: A Target, a Window, and a Reason).
- "This is a compliance exercise." It is a design review of the operational surface. Treating it as compliance is how it becomes box-ticking, and the box-ticking version is genuinely worthless.
- "Only large organisations need this." A three-person team needs the same three answers — who is paged, how do we know, how do we get back — they just need them on one page rather than in a process.
- "We can add the missing pieces later." Some of them are architectural. A rollback story and a migration strategy are much cheaper before launch than after (Expand, Migrate, Contract).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For any production service, you can open its dashboard, its runbook and its ownership record without asking a person.
- The last few incidents on recently-launched services did not begin with "who owns this?" or "how do we roll this back?".
- Reviews produce dated, owned follow-ups, and those follow-ups close.
- A rollback has been performed for this service at least once, deliberately, before it was needed (Rollback: Only Useful If It Is Actually Safe).
- The reversible thing here is the process weight. If the review has become a queue that teams route around, cut it to the three items that genuinely block — owner, detection, recovery — and let the rest be advisory.
- If a service launched without a review and is already carrying traffic, do not roll back the launch. Run the review retrospectively and treat the gaps as prioritised work; a review whose only available answer is "unlaunch" will not be run.
- Automate the checkable dimensions: does an ownership record exist, does an alert route somewhere real, is there a dashboard, are resource requests set, do backups run and has a restore been tested (Restore Drills).
- Automate the collection so the review starts from a pre-filled state and the meeting is spent on the judgement items.
- Do not automate the readiness decision. "Is this safe enough to launch given what it does and who uses it?" is a risk judgement with real stakes (Guardrails, Not Gates).
- Do not automate away the conversation. The value of "what happens when the database is slow?" is that a person answers it out loud and discovers they do not know.
- It costs days of engineering time before launch, against an incident that may not happen. That trade is hardest to argue for on exactly the services where it matters most, because they have deadlines.
- A rigorous review slows launches. A shallow one gives false assurance. There is no setting that avoids both.
- Standardising the checklist across an organisation makes it cheaper to run and less relevant to any specific service, which is the usual cost of a shared artifact.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICWho runs the review, whether it can block a launch, and how many tiers exist are organisational conventions. A startup does this as a checklist in a pull request template; a bank has a named forum with sign-off. The dimensions are the same; the ceremony is not.
- GENERALThe thirteen dimensions below hold across platforms. What differs is the evidence: "health checks configured" is a probe spec on Kubernetes, a target-group health check on a cloud load balancer, and a systemd watchdog on a VM.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how much assurance pre-production testing can provide, and why readiness is about the residue it cannot.