How to Automate Something
Repeated manual task, then understand, then standardise, then automate, then monitor the automation — in that order, because skipping a step moves the failure rather than removing it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
A task keeps being done by hand. What is the correct sequence for turning it into automation that is safer than the hand version?
Manual operational work is slow, inconsistent between people, and impossible to do reliably at three in the morning — but automation written directly from a runbook inherits every misunderstanding in it and executes them faster.
Take the runbook, translate each step into a script, and run the script instead. The steps are already written down, so the translation is mechanical.
A runbook is written for a reader who can notice things. It says "restart the service" and assumes a human will see that the disk is full and stop. The script does not stop.
- A runbook is written for a reader who can notice things. It says "restart the service" and assumes a human will see that the disk is full and stop. The script does not stop.
- Runbooks encode the happy path. The branches a human takes without recording them — checking the dashboard first, noticing the deploy an hour ago, deciding this looks different — are exactly what is missing from the transcription.
- A script derived from an unstandardised task encodes one person's version of it. Everyone else was doing it slightly differently for reasons that are now invisible.
- The script has no observability. It fails silently at step four, having done steps one to three, and nobody finds out until the consequence surfaces (Cron Jobs in Production).
- Nobody owns it. It sits in someone's home directory or an unattributed pipeline, and works until an assumption underneath it changes.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The sequence exists because each step supplies something the next one requires. Understanding tells you which branches exist; standardising collapses those branches to one; automating encodes it; monitoring tells you when the encoding stopped matching reality.
- Skipping *understand* automates a procedure whose purpose nobody can state, so nobody can tell when it becomes wrong (The Automation Trap).
- Skipping *standardise* automates one variant of a task that has several, and the other variants become exceptions handled by hand — which is the toil you were removing, now with an extra system.
- Skipping *monitor* produces the most dangerous shape in this module: automation that has silently stopped working, in a place where its success used to be assumed. Automation is a production system, and an unmonitored production system is one you find out about from users.
- The automation acquires the privileges of the person who used to do the task, exercised more often and with no pause. Its blast radius is therefore at least as large as theirs and its rate is much higher.
The five steps, and what skipping each one produces
The ordering is not a process ritual. Each step is a precondition for the next: you cannot standardise a task whose variations you have not understood, and you cannot monitor an automation whose intended outcome nobody stated.
- 1Notice the repetition
Identify that this task recurs and that its frequency grows with the system (Toil).
fails by Automating a one-off, which costs more than doing it.
evidence You can say how often it happened last month and why.
- 2Understand it
Learn what the task is for, what it assumes, and which branches a human takes.
fails by Transcribing a runbook, inheriting its unstated assumptions (The Automation Trap).
evidence You can explain what each step is for, and what would make it the wrong thing to do.
- 3Standardise it
Collapse the variants to one agreed procedure; discover which variants exist for real reasons.
fails by Encoding one person's version, leaving the rest as manual exceptions.
evidence Two different people, following the written procedure, produce the same result.
- 4Automate it
Encode the standard version: idempotent, bounded, with a dry-run mode and a kill switch.
fails by A script with unbounded scope and no way to stop it mid-run.
evidence A dry run reports exactly the intended changes; a re-run after partial failure converges.
- 5Monitor the automation
Report every run, alert on failure and on absence, and record what changed.
fails by Silent success assumed; a stopped job discovered weeks later (Job Scheduler Reliability).
evidence Deliberately breaking it produces a page, and deliberately preventing it from running produces a different page.
The last step is the one that converts automation from a risk into a reliability improvement. Before it exists, you have replaced a person who would have noticed something odd with a process that will not.
Deciding whether this task should be automated at all
Not everything repetitive is worth automating, and the interesting axis is not frequency alone. It is frequency against how stable the task is and how bad an unattended wrong execution would be.
The best outcome is frequently the one nobody proposes: remove the reason the task exists. A nightly restart script is an excellent automation of a memory leak that should have been fixed.
How should we deal with a recurring operational task?
when The task exists because something upstream is defective — a leak, a misconfiguration, a missing limit.
cost Usually a real engineering fix rather than an afternoon, and it competes with feature work.
when Frequent, well understood, standardised, reversible, and bounded in scope.
cost A production system to own, monitor and keep current as the world changes.
when Understood and standardised, but the judgement of *when* is situational.
cost Still requires a person, so it does not help at 3am unless that person is on call anyway.
when The decision is genuinely human but the data gathering is mechanical.
cost Least glamorous option and often the highest value per hour spent.
when Rare, or irreversible, or a task whose variation carries information a script would discard.
cost It stays slow and inconsistent — which is acceptable when the alternative is fast and wrong.
when The task has three variants and nobody knows why.
cost Delay, and the frequent discovery that two of the variants were nobody's intent.
Automation is a production system
The most common structural mistake is treating automation as a tool rather than as a service. A tool is something you use and observe using. A service runs without you watching, and therefore needs everything a service needs.
Apply the readiness questions you would apply to anything else you deploy — because it is deployed, it acts on production, and it does so with privilege (Production Readiness Review).
| A service needs | The automation equivalent | What its absence looks like |
|---|---|---|
| An owner | A named team, not the person who wrote it | A script in a home directory that stops working after someone leaves |
| Version control and review | The automation is code, reviewed like code | Changes made in place on a production host, unlogged |
| A deployment path | Shipped through the same pipeline as anything else | Copied to a server by hand; nobody knows which version is running |
| Health signals | Ran / finished / duration / what it changed | Silence, which is indistinguishable from success (An Alert Should Demand Action) |
| Alerts | On failure *and* on absence of a run | A job that stopped three weeks ago and was noticed by its consequence |
| A runbook | What to do when it fails halfway | An operator reverse-engineering the script during the incident (Runbooks) |
| Least privilege | Scoped to what it actually needs, per environment | A credential that can change anything, used daily (Identity and Access Management (IAM)) |
| A kill switch | Disable without a deploy | Mitigation requires a pipeline run while the automation keeps acting |
| A rollback story | Reverse the action, or bound it so reversal is feasible | A completed run that cannot be undone and was not meant to happen |
How to do it properly
Most important first.
- Do the task manually enough times to understand what varies and why, and write down the branches — including the ones you took without thinking.
- Standardise before encoding: agree on one way, remove the variants that exist for no current reason, and find out which ones exist for a real reason.
- Automate the standardised version, and make it idempotent so a re-run after a partial failure converges rather than compounds (Job Idempotency).
- Build in a bounded blast radius from the first version: a dry-run mode, a batch limit, a rate limit, and a threshold above which it stops and asks.
- Monitor the automation as a service: did it run, did it finish, how long did it take, what did it change, and alert on *absence* as well as failure (An Alert Should Demand Action).
- Give it an owner, a runbook of its own, and a documented way to disable it without a deploy (Feature Flags: Deploy Is Not Release).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Automation acts with standing privilege and without hesitation, so the default containment is nothing. What contains it is built in deliberately: batch limits, rate limits, a dry-run default, a threshold that stops and asks, and a kill switch that does not require a deploy.
What can go wrong
- Automation that fails silently, in a place where nobody was checking because it used to be a person's job.
- Automation that succeeds at the wrong thing — the steps ran, the outcome was not achieved, and the exit code was zero.
- Automation that becomes stale as the system around it changes, and is discovered stale during the incident it was supposed to handle.
- Automation with broader standing privileges than any individual has, because it needed them once for one step (Least Privilege in Production).
- Automation nobody can turn off quickly, so mitigating it during an incident requires a code change (The Agent Kill Switch).
- Partial automation that leaves the hardest twenty percent manual and the knowledge of how to do it decaying.
- "If it is repetitive, automate it." If it is repetitive *and* understood *and* standardised. Repetitive and not understood is the trap (The Automation Trap).
- "The script ran successfully, so the task is done." Exit codes report step completion, not outcomes. Verify the outcome the task existed to produce (A Successful Deploy Is Not Evidence of a Healthy System).
- "Automation removes human error." It removes per-execution variance and introduces a new class: one error, replicated perfectly at machine speed.
- "We will add monitoring later." Later is after the first silent failure, which is the failure that monitoring existed to prevent.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The automation reports each run: when, what changed, how long, outcome.
- An absence of runs pages someone, rather than being noticed eventually.
- A deliberate dry run shows exactly what would change, and the number is what you expected.
- A person who has never run it can determine, from the record alone, what it did last time.
- Every automated action needs a reverse or a bounded scope. If neither is possible, the action stays behind a deliberate human confirmation (Guardrails, Not Gates).
- Disabling must be possible without a deploy — a flag, a schedule pause, a queue drain — because the moment you need to stop it is not the moment to be waiting on a pipeline.
- Where the automation acts in batches, keep the batch small enough that reversing one batch is a manageable operation.
- Automate the mechanical and the well-understood: the steps whose correctness does not depend on what else is happening at the time.
- Do not automate the decision about whether the situation calls for this action at all. Encoding "restart when memory is above X" is fine; encoding "decide whether restarting is the right response" is not (Stop the Harm Before You Understand It).
- Automate the recording of what happened before automating the doing of it. Observability first is cheap, and it is what makes the automation reviewable later.
- Automation is code with an owner, a test surface and a maintenance cost, traded against work that used to be absorbed by people.
- Automating a task removes the practice that kept people able to do it by hand, which matters when the automation is unavailable (Break-Glass Access).
- Standardising first is slower and frequently reveals that the task should not exist, which is a better outcome and does not feel like progress.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe sequence holds for any operational task on any stack. What changes with context is how much of the standardise step is already done for you — a fleet built from one template needs almost none, and a fleet of hand-built systems may need more standardisation work than the automation itself.
- ORG-SPECIFICHow much autonomy automation is granted is a local risk decision. Some organisations allow automated remediation to restart, scale and fail over unattended; others require every production-mutating action to be initiated by a person. Both are defensible, and the difference is risk appetite, not maturity.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.