Change Management
Deciding which changes need what scrutiny, so that ordinary changes stay cheap and genuinely risky ones get attention — without a process people route around.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Which changes should require approval, from whom, and how do we avoid a process that adds delay without adding safety?
Some production changes genuinely warrant scrutiny and most do not. A single policy for both either gates everything, making delivery slow and encouraging bypass, or gates nothing, so the irreversible changes get the same attention as a typo fix.
All production changes go through the approval process. It is uniform, it is auditable, and nobody can claim they did not know the rules.
Uniform gates price ordinary changes at the cost of dangerous ones, so people batch to amortise the process — and batching is what makes changes dangerous (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Uniform gates price ordinary changes at the cost of dangerous ones, so people batch to amortise the process — and batching is what makes changes dangerous (Change Size: Why Small Changes Are Safer, and When They Are Not).
- An approver who cannot evaluate the change approves it anyway. The approval becomes a signature, which is auditable and not protective (Review as a Gate).
- Slow processes create bypass paths, and bypass paths get used under pressure, so the least-reviewed changes are the ones made in the worst conditions (Manual Production Changes).
- Uniform process treats an additive config change and a destructive migration as the same risk, which trains everyone to ignore the classification (Destructive Migrations).
- Approval queues add latency between writing a change and observing its effect, which is the feedback loop the whole discipline exists to shorten (What DevOps Actually Means).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Change management is risk-proportionate scrutiny: matching the amount of review to the reversibility and blast radius of the change, rather than applying one rule to all of it.
- Two properties do almost all the classification work: can it be undone, and how many are affected if it is wrong (Blast Radius: If This Is Wrong, How Much Does It Affect?). A reversible change with a small radius needs a peer review and nothing else; an irreversible change affecting everyone needs a named decision-maker.
- The most effective control is usually not a gate but a guardrail — making the safe path the easy one, so the dangerous form is hard to express rather than merely disallowed (Guardrails, Not Gates).
- Gates are worth their cost in exactly one place: changes with no rollback. That is a short list — destructive migrations, data deletion, irreversible external side effects, credential and access changes.
- Emergency changes need a defined path rather than an exception, because they will happen. The path skips scrutiny in exchange for a mandatory reconciliation afterwards (Break-Glass Access).
- Regardless of process weight, every change needs a record: what, who, when, why, and what the previous state was. The record is separable from the approval, and it is the part that is genuinely non-negotiable (The Audit Trail).
Classify by reversibility and radius
These two axes decide almost everything. Note that the fourth row is the only one where a gate clearly earns its cost — and that it is also the smallest category.
| Reversible? | Blast radius | Examples | Proportionate scrutiny |
|---|---|---|---|
| Yes, in minutes | Small or progressive | Feature flag ramp, config value with a canary, routine deploy | Peer review at merge; automated verification; no gate |
| Yes, in minutes | Everyone at once | Global config change, a flag with no cohorting, cache invalidation | Peer review plus a progressive rollout requirement — reduce the radius rather than adding a gate (Reducing Blast Radius) |
| Yes, slowly | Everyone | Infrastructure change requiring re-provisioning, DNS change with TTL propagation | Peer review, a plan output someone reads, and a stated recovery time (The Plan: Desired vs Current) |
| No | Everyone | Destructive migration, data deletion, credential rotation with no overlap, irreversible external side effect | Named approver, verified backup, explicit record. The one place a gate is clearly worth its cost (Destructive Migrations) |
| Yes | One tenant or one cohort | Tenant-scoped config, targeted flag | Peer review; the radius is already the containment (Multi-Tenancy) |
| Varies, under pressure | Everyone | Incident mitigation | A defined emergency path: act now, record now, reconcile after (Break-Glass Access) |
Where the scrutiny should sit
Given a change that needs more than peer review, there are several places to put the additional control, and they are not equivalent — some add safety, some add only a record.
This change class needs more than automated verification. What do you add?
when The risk has a detectable shape: a destructive statement, an over-broad IAM grant, a missing lock timeout.
cost Requires the rule to be encoded and maintained; false positives block legitimate work (Policy as Code).
when The risk is comprehensible from the diff by someone with domain context.
cost Depends on reviewer availability and expertise; degrades quietly when both are scarce (Review as a Gate).
when The change is irreversible and someone specific should own the decision.
cost Latency and queueing, plus the risk of becoming a signature rather than an evaluation.
when The change is reversible but currently affects everyone at once.
cost Engineering work to add cohorting or progressive rollout — usually a better investment than a gate (Progressive Delivery: Exposure as a Dial).
when Safety depends on a precondition: a tested restore, a completed backfill, a soak period.
cost Evidence must be checkable automatically, or it becomes a checkbox (Restore Drills).
when The change is reversible with a small radius and the real need is traceability.
cost Almost none, which is why it should be the default everywhere (The Audit Trail).
The emergency path
Every process needs one, and its design decides whether it stays exceptional. The rule that keeps it honest is that it trades scrutiny for reconciliation rather than skipping both.
- 1Invoke explicitly
The operator declares they are using the emergency path.
fails by Silent bypass, so nobody knows the normal controls were skipped.
evidence An explicit invocation exists, with a person's name on it.
- 2Notify in real time
Announces the change to the incident channel as it happens.
fails by Discovering an unannounced production change while diagnosing something else (Telling People What Is Happening).
evidence The change appears in the incident timeline at the time it happened.
- 3Act
Makes the change, with normal gates skipped and elevated access if needed.
fails by Access that has to be requested during the incident, adding minutes to a mitigation (Break-Glass Access).
evidence The mitigation took effect, verified by the signal that detected the problem.
- 4Record automatically
Captures what changed, by whom, when, and the previous state — without asking the operator to write it.
fails by Relying on someone to document it afterwards, from memory, at 4am (The Audit Trail).
evidence A complete audit entry exists without anyone having typed it.
- 5Reconcile
Brings the emergency change back onto the normal path: committed, reviewed, and reflected in the desired state.
fails by Never happening, so the next deploy silently reverts the mitigation (Drift).
evidence The change exists in the repository and the running state matches it.
- 6Review the invocation
Asks, afterwards, whether the emergency path was needed — and if it was needed because the normal path is too slow.
fails by Treating frequent use as normal rather than as a signal (Postmortems).
evidence Invocation frequency is tracked and discussed.
The reconciliation step is the one that decides whether this stays an emergency path. Without it, the fastest way to change production is also the way that leaves no lasting record, and people are not wrong to prefer it.
How to do it properly
Most important first.
- Classify by reversibility and blast radius, and state the classification in the pipeline rather than in a document nobody opens (Policy as Code).
- Make the ordinary path — reversible, small radius, verified automatically — require peer review at merge and nothing else.
- Reserve approval gates for irreversibility. If a change can be undone in minutes, an approval is buying very little.
- Automate detection of the high-risk classes so classification is not a self-declaration: destructive SQL, IAM changes, deletions in infrastructure plans (Destructive Changes: What a Rename Really Does).
- Define the emergency path in advance: who can invoke it, what it skips, who is notified in real time, and what must be reconciled afterwards.
- Measure whether the process is being bypassed. Frequent bypass is information about the process, not about the people (Toil).
- Record every change regardless of which path it took (The Audit Trail).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The process itself is the containment for the classes it covers. It contains nothing it does not cover, which is why misclassification is the failure mode that matters most.
What can go wrong
- Approval theatre: an approver without the context to evaluate the change, producing an audit record and no scrutiny.
- Classification by self-declaration, so the risky change is labelled routine by the person who wants it to ship.
- An emergency path that is faster and easier than the normal one, which makes it the default (Break-Glass Access).
- A blanket freeze period that batches a fortnight of changes into the first day after it lifts, concentrating risk exactly where the freeze was trying to reduce it.
- Gates that block during incidents, so mitigations are delayed by a process designed for planned change.
- The mitigation failing: automated classification that misses a class — an ORM-generated destructive statement, a Terraform replace that is really a delete and recreate — so a gate that everyone trusts silently does not cover the case (Destructive Changes: What a Rename Really Does).
- "More approval means more safety." Approval by someone who cannot evaluate the change adds a signature. Safety comes from reversibility, small blast radius and fast detection — approval is a weak substitute for all three.
- "Change management is bureaucracy." Bureaucracy is one implementation. The underlying question — which changes deserve scrutiny — has to be answered by everyone, including teams who think they have no process.
- "A freeze reduces risk." It defers risk and concentrates it. Sometimes correct for a genuine high-stakes window; frequently a way to move an outage to a less convenient date.
- "We are regulated, so we must gate everything." Regulation typically requires evidence, separation of duties and traceability. Automated evidence and review at merge satisfy those more reliably than a manual approval queue (The Audit Trail).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The proportion of changes needing an approval is small and stable, and the classes requiring one are written down.
- Bypass rate is measured, and is low — and when it rises, the process is examined rather than the individuals.
- For every production change in the last month there is a record of who, what, when and why, regardless of path.
- An approver can, when asked, state what they were checking for. If the answer is "that it looked reasonable", the gate is theatre.
- The emergency path has been used and the reconciliation happened within a stated window.
- Process changes are reversible and should be reviewed on a cadence, because processes accrete: each incident adds a control and nothing removes one.
- The most valuable thing change management contributes to rollback is knowing, per change class, whether rollback exists at all — which is why reversibility is the primary classification axis rather than a secondary attribute.
- A freeze is a change-management decision with its own rollback cost: whatever is held back arrives together when it lifts (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Automate classification — detecting destructive migrations, privilege changes and infrastructure deletions is far more reliable than asking authors to self-assess (Policy as Code).
- Automate the record, so an audit entry is a side effect of the change rather than a task.
- Automate the routine path end to end, so the gates that remain are rare enough to take seriously.
- Keep human: the approval itself for irreversible changes, and the periodic review of whether the classification still matches reality (The Automation Trap).
- Every gate costs latency on the changes it covers, and latency is paid by every change including the safe ones caught by an imprecise rule.
- Risk-proportionate process is harder to explain to an auditor than a uniform one, and the classification itself becomes something that must be justified.
- Automated classification has false positives, and each one is a person blocked by a rule that is wrong about their change.
- Fewer gates means more reliance on automated verification, which shifts the investment rather than removing it (Continuous Deployment).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICChange classes such as standard, normal and emergency come from IT service management frameworks and are conventions, not facts. Who approves, what a change record must contain and whether an advisory board exists vary by organisation and regulator. What generalises is only the classification axes — reversibility and blast radius — and the requirement that a record exists.
- GENERALEvery team has change management, including teams that have never used the phrase. The question is only whether the rules were chosen or defaulted into.
- SCALE-SPECIFICAt one team, peer review plus a rule about destructive changes is a complete implementation. Explicit classification, approver roles and formal records become necessary when the person making a change and the person who will be paged for it are routinely different people.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.