Manual Production Changes
Sometimes an emergency requires acting by hand. The damage is not the manual change — it is the permanent, undocumented divergence between production and the code that is supposed to describe it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
An emergency required changing production by hand. What now?
Every reviewed, automated path has a failure mode where it is unavailable or too slow, and the incident is happening now. Someone acts by hand, correctly. Then the incident ends, everyone sleeps, and production quietly no longer matches the code that claims to define it.
Never touch production manually. Everything goes through the pipeline, without exception.
The pipeline is sometimes what is broken. A CI outage, a broken deploy path, an expired credential in the release job — insisting on the pipeline means the outage continues while you fix the tooling (CI/CD Anti-Patterns).
- The pipeline is sometimes what is broken. A CI outage, a broken deploy path, an expired credential in the release job — insisting on the pipeline means the outage continues while you fix the tooling (CI/CD Anti-Patterns).
- Some emergencies have no automated path at all: repairing corrupted rows, clearing a poisoned cache entry, forcing a stuck reconciliation (Dead Letter Queues Are an Operation).
- An absolute prohibition that people violate under pressure is worse than an honest procedure, because the violation is then also concealed. You lose the change *and* the record of it.
- The pipeline can be slower than the incident allows. A twenty-minute deploy during a ten-minute-per-hour revenue loss is a defensible thing to bypass, and the policy should say so rather than being routed around.
- Prohibition without a documented alternative teaches people that the honest path is punished, which is exactly the norm that makes drift permanent.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A manual change creates drift: production state that no code describes. Drift is not immediately harmful — the system is working, that was the point — and that is precisely why it survives (Drift).
- The damage arrives later, in three specific ways. The next deploy silently reverts the manual fix and the incident recurs, apparently spontaneously. Or the environment is rebuilt and the fix is not, so the new environment is subtly different. Or someone reads the code, reasons correctly from it, and is wrong about production (Configuration Drift).
- The third one is the worst because it is unbounded. Once code cannot be trusted to describe production, every subsequent decision made by reading the code is suspect, and nobody knows which ones.
- The correct model is therefore not prohibition but a debt with a repayment schedule. The manual change is a legitimate emergency instrument; the reconciliation back into code is the obligation it creates, and it must be discharged in days, not "when we get to it".
- Reconciliation has two acceptable endings and one unacceptable one. Either the change is committed into code so the state is described, or it is reverted because it was a temporary measure. Leaving it in place, undescribed, is the failure.
- Some platforms make drift self-correcting: a reconciliation loop reverts unmanaged changes on its own, which converts silent permanent drift into a loud immediate revert. That is a better failure mode, and it is also a trap if you did not know it would happen (Reconciliation: The Loop Under Everything).
What the manual change costs, and when
The reason drift survives is that the moment of the change is the moment of maximum justification and minimum consequence. Everything that goes wrong goes wrong later, to someone else, in a form that does not look connected.
Read the timeline as three separate future incidents, all seeded by the same unreconciled minute.
- Day 0, 03:12changeConnection pool exhausted. On-call raises
max_connectionsfrom 50 to 200 directly on the running configuration. Impact ends in ninety seconds. - Day 0, 03:20recoveryIncident closed. The change is mentioned in the channel and not written down anywhere else.
- Day 0, 09:00actionEveryone sleeps. The reconciliation ticket that was going to be opened is not opened.
- Day 4, 11:30changeRoutine deploy. The config in the repository still says 50. The deploy applies it. Nobody notices.
- Day 4, 14:15signalConnection pool exhausted again, under ordinary load. The incident appears to have no trigger — no code change explains it.
- Day 4, 14:40actionDiagnosed by an engineer who remembers the earlier incident. Value raised again, by hand, again. Still not reconciled.
- Day 18actionA new engineer reads the repository, sees 50, and sizes a downstream capacity estimate against it. The estimate is wrong by a factor of four (The Connection Budget).
- Day 31signalA second region is stood up from code. It gets 50 and fails under a fraction of the traffic the primary handles.
- Day 44signalDrift detection is finally added. It reports thirty-one differences between declared and actual state, of which nobody can say which were deliberate.
The pattern is that each consequence looks like a new problem. Nobody at day 18 or day 31 has any reason to connect what they are seeing to ninety seconds of correct emergency work six weeks earlier — which is exactly why the reconciliation has to happen in days, while the connection is still obvious.
The procedure
Emergency manual change is a legitimate operation with an obligation attached. Treating it as a procedure — rather than as a policy violation people commit quietly — is what makes the obligation collectible.
- 1Decide
Establish that the reviewed path is unavailable or too slow for the impact in progress.
fails by Chosen out of habit because it is quicker, when the pipeline would have worked.
evidence A stated reason: pipeline down, or impact rate versus deploy duration.
- 2Announce
Say in the incident channel what is about to be changed and where.
fails by Done silently, so others debug against assumptions that are no longer true.
evidence A timestamped message before the change (Telling People What Is Happening).
- 3Capture
Record the current value or state before changing it.
fails by Skipped under pressure, making the change irreversible in practice.
evidence The previous state pasted into the incident channel.
- 4Change
Apply the smallest change that stops impact.
fails by A broad change that is harder to reconcile than the problem warranted.
evidence One value, one resource, one table — scoped and recorded.
- 5Verify
Confirm impact has ended, using the signal that detected it.
fails by Confirmed by the change having been applied rather than by impact ending.
evidence The detecting signal back within its normal band (A Successful Deploy Is Not Evidence of a Healthy System).
- 6Ticket
Open the reconciliation task now, during the incident.
fails by Deferred to the morning, which does not arrive with the same urgency.
evidence A ticket referencing the incident, created before the incident is closed.
- 7Reconcile
Commit the change to code, apply through the normal path, or revert it deliberately.
fails by Left indefinitely; the code no longer describes production.
evidence A merged change, applied through the pipeline, with production state unchanged by the apply.
- 8Detect
Drift detection confirms declared and actual state now agree.
fails by No detection, so the next unreconciled change is equally invisible.
evidence A clean drift report, or a report whose remaining entries are each explained (Drift).
The Ticket step is the whole procedure in miniature. It costs thirty seconds during the incident and it is the only step that survives everyone going to sleep.
Concealment is the expensive failure
The two versions below describe the same emergency and the same correct technical decision. What differs is whether anybody else can know about it, and that difference is worth more than the change itself.
A policy that produces the left-hand column is not a stricter policy. It is a policy that has traded a known debt for an unknown one.
The pipeline was down, so an engineer edited the running config directly to raise the connection limit. Impact ended. Nobody wrote it down, because manual production changes are against policy and mentioning it invites a conversation nobody wants. Four days later a routine deploy reverted the value and the incident recurred, apparently from nowhere. The second incident took longer than the first, because this time there was no obvious trigger to find.
The pipeline was down, so an engineer posted "raising `max_connections` on checkout-db from 50 to 200, pipeline is down, previous value recorded" in the incident channel and made the change. Impact ended. A reconciliation ticket was opened before the incident was closed. The next morning the value was committed to the repository and applied through the normal path; the apply changed nothing, confirming the two now agree. The postmortem lists the pipeline outage as a contributing factor to incident duration.
The second version costs about two minutes more, in total, across the whole incident. It buys: a system whose code describes it, a deploy that does not silently revert the fix, a second engineer who can read the repository and be right, and a postmortem that identifies the pipeline outage as something worth fixing. Concealment bought none of that and did not even avoid the conversation — it just moved it to a worse moment.
How to do it properly
Most important first.
- Say explicitly that manual changes are permitted in emergencies, and make the honest path clearly better than the concealed one. A policy people can follow beats a policy they must break.
- Announce it as it happens, in the incident channel: what you are changing, where, and why. Contemporaneous is worth ten times reconstructed.
- Capture the before state — the current value, the config, the row contents — before changing it. It costs seconds and it is the only thing that makes the change reversible (The Plan: Desired vs Current).
- Prefer the smallest change that stops impact. A scaled replica count is easier to reconcile than a hand-edited config file, which is easier than a hand-written data repair.
- Open the reconciliation ticket during the incident, not after. The intention to do it later does not survive the sleep that follows.
- Reconcile within days: commit the change to code, apply through the normal path, and confirm the pipeline produces the same state (Infrastructure as Code).
- Detect drift automatically so a manual change that was never reconciled surfaces on its own rather than during the next incident (Drift).
- Record it in the postmortem as a fact about the incident. It is timeline material and it is often the most useful line in the document (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The change itself is usually scoped to what the operator touched. The drift it leaves is contained by drift detection and reconciliation — and by nothing at all if neither exists.
What can go wrong
- The change is made, works, and is never reconciled. This is the default outcome without a forcing mechanism.
- Reconciliation attempted from memory a week later, producing code that does something slightly different from what production is doing.
- The next deploy reverts the fix and the incident recurs, with everyone confused because "we fixed that".
- A manual change made through the console that the platform then reverts on its own reconciliation cycle, minutes later, mid-incident (Reconciliation: The Loop Under Everything).
- The change made with elevated access and never mentioned outside the audit log, so the postmortem describes an incident that ended for reasons it cannot explain.
- Drift detection that alerts on every trivial difference until it is muted, at which point real drift is invisible (Alert Fatigue).
- A policy of prohibition that pushes manual changes into concealment, which converts a manageable debt into an unknown one.
- "Manual changes are always wrong." They are sometimes the fastest way to stop user impact, and stopping impact is the priority (Stop the Harm Before You Understand It). The wrong part is leaving them undocumented.
- "We reconciled it because we wrote it in the postmortem." A description is not code. Reconciliation means production state is described by something that would recreate it.
- "Infrastructure as code means drift cannot happen." IaC describes intended state. Nothing stops someone editing production directly unless the platform actively reconciles, and most do not (Infrastructure as Code).
- "The change is small, so it does not need reconciling." Small undocumented changes are the ones that are forgotten and the ones that make the code untrustworthy. Size is unrelated to the harm.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every manual production change in the last quarter has a corresponding commit, or a recorded decision to revert it.
- Drift detection runs and its findings are acted on rather than muted.
- A rebuilt environment matches production, because everything in production is described somewhere (Parity That Is Worth Paying For).
- Incident timelines name manual changes explicitly, with timestamps.
- No incident in recent memory was caused by a deploy reverting an unreconciled manual fix.
- Reverting a manual change requires knowing the previous state, which is why capturing it first is the practice. Without that capture, the rollback is a reconstruction and may be wrong.
- If a manual change is reconciled into code incorrectly, the next apply produces a third state that matches neither. Verify after reconciling: apply through the pipeline and confirm the result equals what production is actually doing.
- Where the platform reconciles automatically, "rollback" may happen without you: the manual change disappears on the next reconciliation cycle. Know whether your platform does this before relying on a manual change to hold (Immutable Infrastructure).
- Automate drift detection across infrastructure, configuration and access. It is the only mechanism that finds unreconciled changes without depending on anyone remembering.
- Automate capture: an audited session that records commands and their output turns "what did we change?" from an archaeology problem into a query (Audit Logs for Privileged Actions).
- Automate the reminder — an open reconciliation ticket created at the moment elevated access is used, closed only by a merged change.
- Do not automate the reconciliation itself. Deciding whether the emergency change should become permanent or be reverted is a judgement about intent, and importing production state into code blindly encodes mistakes as intentions.
- Allowing manual changes creates drift risk. Forbidding them creates incident-duration risk and concealment risk. The second pair is worse, but the first is real and has to be actively managed.
- Capturing state and announcing during an incident costs a minute at the moment when a minute is expensive. It is nearly always worth it and it will not feel that way at the time.
- Drift detection is noisy on most real platforms, and tuning it is ongoing work. Untuned, it gets muted, and a muted detector is worse than none because it is believed to be working.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALDrift from manual change is universal. What differs is how fast it surfaces: a platform with active reconciliation reverts it within minutes, while a hand-managed VM fleet can carry an undescribed change for years.
- ORG-SPECIFICWhether manual changes require approval, and how quickly reconciliation is expected, are conventions. Regulated environments may require a change record before the fact even in emergencies — in which case the record is created during the incident, not instead of acting.
- KUBERNETES-SPECIFICA controller continuously reconciles observed state towards declared state, so a manual
kubectl editis typically reverted at the next sync. That is a better failure mode than silent drift and a trap if you expected the change to persist — the equivalent edit on a VM would have held indefinitely.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — declared state as the source of truth, and what it costs when the running system and its description disagree.