The Automation Trap
Do not automate what you do not understand. Bad automation does not make mistakes less likely — it makes them faster, wider and more confident.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What exactly goes wrong when you automate a procedure you have not understood?
Automation is presented as a way to reduce human error, so the reflex after an incident caused by a manual mistake is to automate the manual step — which removes the person who would have noticed the situation was unusual.
A person did it wrong. Encode the correct procedure so it is done the same way every time and the error class disappears.
The error class does not disappear; it relocates. Instead of one wrong execution you get one wrong *encoding*, applied perfectly to every future execution.
- The error class does not disappear; it relocates. Instead of one wrong execution you get one wrong *encoding*, applied perfectly to every future execution.
- The person doing it by hand had a slow feedback loop attached to their eyes. They noticed the count looked wrong, the environment name looked odd, the list was longer than usual. The automation has no such loop unless someone built one.
- Automation acts at machine speed and machine breadth. A human deleting resources one at a time has minutes of opportunity to stop; a loop has none.
- Automation is trusted. When a script says it succeeded, people believe it in a way they would not believe a colleague who said "I think I did all of them".
- The procedure encoded today is correct against today's system. Nothing tells the script when the assumption it depends on stops holding, so it keeps executing a procedure that has quietly become wrong (Configuration Drift).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Automation changes three variables at once, and only one of them is the one people intend: it removes variance, it multiplies rate, and it removes the observer.
- Removing variance is the intended benefit and it is real. Multiplying rate means a mistake propagates faster than any human process can interrupt. Removing the observer means the class of problem that used to be caught by someone thinking "that does not look right" is no longer caught at all.
- The net effect on risk depends entirely on whether the encoded understanding was correct. Correct understanding automated is a large win. Incorrect understanding automated is a larger loss than the manual version, because manual execution has an accidental rate limit and an accidental reviewer.
- This is the operational form of the Engineer Atlas position: automate the work, never the understanding. A tool that performs a procedure on your behalf is leverage; a tool that means nobody on the team can any longer say what the procedure is for is a liability that has not yet been triggered.
- The same argument now recurs with agents and generated operational code. Delegating the typing is fine. Delegating the comprehension means nobody can evaluate whether the action is correct, and the confidence of the output is not evidence about the system (Deploying an Agent).
The same mistake, by hand and by loop
The mistake is identical in both columns: a filter that matches production resources when it was meant to match a staging set. What differs is how much of it happens before anyone can intervene, and whether anything about the situation prompts a second look.
Job runs on schedule, unattended
-> queries for matching resources
-> filter matches 3,000, not the usual 12
-> no threshold check, so it proceeds
-> deletes all of them in ninety seconds
-> exits 0, logs "cleanup complete"
-> first signal is a service failing
to start, twenty minutes laterEngineer runs the query
-> sees 3,000 rows where they expected 12
-> stops, because that is obviously wrong
-> checks the filter, finds the bug
-> nothing was deleted
(and if they had not noticed:
deletes one at a time, sees the first
few names look wrong, stops at ~5)Nothing about the automated path is smarter or dumber than the person — it encodes the same intent. What it lacks is the two accidental safety properties of manual work: a rate slow enough that a wrong result can be interrupted, and a human looking at the output of each step. Both can be rebuilt deliberately, as a threshold check and a dry-run diff, and neither appears unless someone builds it.
Three seconds per resource is not a small difference
The timeline below traces the automated column above. It is an illustration of the shape rather than a report of a specific incident, and the point of the numbers is the ratio: the destructive phase is over long before the detection phase begins.
Note where the time actually goes. The damage takes ninety seconds; the recovery takes hours, and most of that is spent determining what was lost rather than restoring it.
- T-3wchangeEnvironment tagging convention changes as part of an unrelated migration. Nothing references the cleanup job in that change.
- T-0actionScheduled cleanup runs, as it has every night for a year.
- T+2ssignalFilter now matches production resources as well as the intended staging set. No threshold check exists to notice the count changed by two orders of magnitude.
- T+90sactionDeletion complete. Exit code 0. The log line reads "cleanup complete".
- T+20msignalA deploy fails: a resource it expected is gone. Treated as a deploy problem.
- T+35msignalSecond and third services report missing dependencies. An incident is declared.
- T+50mactionSomeone correlates the timing with the nightly job. The job is disabled — first mitigation, before anyone understands the filter (Stop the Harm Before You Understand It).
- T+1h10mactionRestore begins. The bounded part: what was backed up comes back. The unbounded part: determining what existed at all, since the inventory was partly derived from the resources themselves.
- T+5hrecoveryService restored. Some derived state is rebuilt rather than recovered.
- T+1dchangeFix is not "be more careful with filters". It is: dry-run diff by default, refuse if the match count deviates from the trailing norm, delete in batches with verification between them, and soft-delete with a retention window.
The trigger was a tagging change three weeks earlier that nobody connected to the job — the assumption the automation depended on was invalidated by a change that had no reason to know the automation existed. Assertions inside the automation are what catch that; documentation of the assumption is what does not.
What makes automation dangerous, and what to do about each
Four properties turn a helpful script into an incident, and each has a specific, cheap countermeasure. None of the countermeasures is "be careful"; all of them are code.
- A dry-run mode that prints the diff is the highest value-per-line safety feature available, and it is usually a day of work.
- A threshold check — "if this run would affect more than N, stop and ask" — catches the entire class of wrong-filter failures without needing to know which filter is wrong.
- Assertions beat documentation. "This job assumes environment tags follow convention X" in a README is invisible to the person who changes convention X; the same statement as a check that fails loudly is not.
- Soft delete with a retention window converts the worst automated failure class from irreversible to inconvenient.
| Property | Why it helps | Why it hurts | The countermeasure |
|---|---|---|---|
| Speed | The work finishes in seconds instead of an afternoon | A mistake also finishes in seconds, with no window to interrupt | Rate limit; batch with verification between batches |
| Breadth | One run covers the whole fleet | One mistake covers the whole fleet | Hard cap on scope per run; refuse when the match count deviates from the norm |
| Privilege | It can complete the task without asking | Its worst case is the worst case of its credentials | Scope per environment; separate read and mutate identities (Least Privilege in Production) |
| Confidence | Consistent, no fatigue, no drift between operators | A green exit code is believed more readily than a colleague would be | Verify the outcome, not the exit code (A Successful Deploy Is Not Evidence of a Healthy System); log what actually changed |
| No observer | Nobody has to watch it, which is the point | Nobody notices the situation is unusual | Assert the assumptions in code; alert on anomalous action volume |
How to do it properly
Most important first.
- Require that someone can state, before it is automated, what the procedure is for, what it assumes, and what would make it the wrong action.
- Build the blast-radius limit into the first version, not the second: dry run by default, an explicit batch size, a rate limit, and a refusal above a threshold (Reducing Blast Radius).
- Make the automation assert its assumptions and stop when they do not hold, rather than proceeding through an unexpected state.
- Make it explain itself. An action log that says what was matched, what will change and why is what lets a human re-acquire the observer role.
- Keep a kill switch that works without a deploy, and rehearse using it (The Agent Kill Switch).
- Automate the reversible first. If the action cannot be reversed, either bound it hard or keep a human at the final step (Guardrails, Not Gates).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Frequently nothing, and that is the lesson — automation acts faster and wider than a human would, so its worst case is bounded only by what it was permitted to touch. Containment must be built in: batch size, rate limit, assertion checks, dry-run default, kill switch.
What can go wrong
- A selector or filter that matches more than intended, and a loop that acts on all of it before anyone can react.
- An assumption that held when the automation was written and quietly stopped holding — an environment renamed, a tag convention changed, an API returning a paginated result the script only read the first page of.
- A safety mechanism that was tested against the success path only, so the abort logic is itself untested when it is needed.
- Automation running with standing broad privilege because it needed it once, making its worst case the account's worst case (Least Privilege in Production).
- Retry logic layered on an action that is not idempotent, converting one partial failure into repeated partial effects (Job Idempotency).
- A team that no longer knows how to perform the procedure manually, discovering this while the automation is the thing that is broken.
- "Automation reduces human error." It relocates human error from execution time to design time, where it is made once and applied every time. That is an improvement only if the design was right.
- "It has run a thousand times without incident, so it is safe." It has been correct for a thousand inputs that resembled each other. The relevant question is what it does with an input that does not.
- "The automation is the cause of the incident, so we should go back to doing it manually." Usually the fix is bounds and assertions, not reversion. But going back is a legitimate option, and refusing to consider it because it feels backwards is how teams keep a dangerous script.
- "The model generated it and it looks right." Generated operational code is a draft with high fluency and no accountability. The reviewer supplies the understanding; if nobody does, nobody has (Prompts and Models Are Deployables).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Someone who did not write the automation can explain what it does and when it would be wrong.
- A dry run against production produces a list a human reads, and the size of the list is unsurprising.
- A deliberately unexpected input causes it to stop rather than continue.
- The kill switch has been used at least once, in a drill.
- For destructive automation there is often no rollback, which is precisely why the containment must be in the scope rather than in the recovery: act on a batch, verify, then act on the next.
- A soft-delete or retention window converts an irreversible automated action into a reversible one, and is usually the single highest-value change you can make to a cleanup job (Partial and Logical Data Recovery).
- When automation is the cause of an incident, disabling it is the first mitigation, before diagnosis. It is still acting while you think (Stop the Harm Before You Understand It).
- Automate execution; keep comprehension. The test is whether a person can still answer what the automation does and why, without reading it line by line during an incident.
- Do not automate a procedure whose failure mode nobody has articulated.
- Do automate the guardrails around your other automation — dry-run diffing, threshold checks and action logging are themselves mechanical, and they are the parts that let the rest be trusted.
- Every safety mechanism — dry runs, batch limits, assertions, confirmations — makes the automation slower and more code, which is the whole cost and it is worth paying.
- Insisting on understanding before automating is slower than transcribing a runbook, and it regularly reveals that the procedure is wrong, which delays the automation and improves the system.
- Keeping people able to do the task manually costs practice time that automation was supposed to save.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe rate-and-breadth argument is arithmetic and holds anywhere: a person performs an action at human speed with an accidental review at each step, and a loop does not. Nothing about a particular stack changes that.
- ORG-SPECIFICHow much unattended authority automation gets is a risk decision. A team with fast, tested restore paths can afford more autonomous action than a team whose recovery from a bad bulk operation is a support ticket to a provider — the same script is a reasonable tool in one and an unacceptable risk in the other.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.