Toil
Manual, repetitive, automatable operational work that scales with the service and leaves nothing behind — and the "scales with the service" part is what makes it toil rather than just work.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Which operational work is genuinely worth engineering away, and which work only feels like it should be?
Operational load grows with the number of services, tenants and requests, so a team that absorbs it by working harder discovers that its capacity for engineering shrinks exactly as the system that needs engineering grows.
Operational work is part of the job. Absorb it, and automate whatever gets annoying enough that someone complains.
The work grows with the system while the team does not. Two hours a week per service is fine at three services and is a full-time job at thirty.
- The work grows with the system while the team does not. Two hours a week per service is fine at three services and is a full-time job at thirty.
- It is absorbed invisibly. Nobody records the twenty minutes spent restarting something, so the growth is not visible until people are visibly overloaded.
- "Annoying enough to complain about" selects for irritating work, not for expensive work. The genuinely expensive toil is often the routine that everyone has stopped noticing.
- The team's remaining engineering capacity is spent on incidents, so the underlying causes of the toil never get fixed and the toil keeps growing (Rotations People Can Sustain).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The standard definition has six properties, and the discriminating one is the last: toil is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth.
- That last property is what separates toil from ordinary work. A one-off manual migration is manual, repetitive within itself, and tedious — and it is not toil, because doing it once removes it. Toil is the work that arrives again next month, and more of it than last month.
- The economics follow directly. Work that does not scale with growth can be absorbed; work that does compounds, and absorbing it is a strategy with a fixed lifespan.
- "Devoid of enduring value" means the system is in the same state afterwards as before. Restarting a service returns it to where it was. Fixing the leak that made it need restarting changes the system permanently — that is project work, and it is what reduces future toil.
- Some toil is irreducible and some is a symptom. The distinction matters more than the measurement: automating a symptom freezes the defect underneath it in place, and makes it cheaper to keep.
The six-property test
Run a concrete piece of work through all six properties. The point of the test is not to produce a label; it is that the property which *fails* tells you what response the work actually needs.
The most useful row is the last. Work that does not scale with growth can be absorbed indefinitely; work that does is a commitment that gets larger every quarter.
| Property | Asks | If this is false, the work is... |
|---|---|---|
| Manual | Does a human have to perform it? | Already automated — the question is whether the automation is monitored (How to Automate Something) |
| Repetitive | Has it been done before, the same way? | Novel: it is project work, or an incident, and deserves attention rather than a script |
| Automatable | Could a machine do it, with the same outcome? | Judgement work: routing an incident, deciding a rollback, sizing capacity. Support it with tooling, do not replace it |
| Tactical | Is it interrupt-driven rather than planned? | Planned engineering. It may be tedious; it is not toil |
| No enduring value | Is the system in the same state afterwards? | A permanent improvement. That is exactly the work that reduces future toil |
| Scales with growth | Does more service mean more of it? | A one-off. Do it, be annoyed, move on — automating it costs more than it returns |
Automate the task, or remove the reason for the task
The most consequential decision about a piece of toil is made before any code is written, and it is frequently made by default: writing the script is faster than fixing the cause, so the script gets written.
That is sometimes right. What makes it dangerous is that a good script removes the pain that would otherwise have forced the fix, and the defect underneath becomes permanently affordable.
Memory grows until the process is killed
-> add a scheduled nightly restart
-> pain disappears, ticket closed
-> leak still there, now invisible
-> six months later the leak accelerates
-> nightly is not enough; restart hourly
-> a restart lands mid-request during
peak traffic, and now it is an incidentMemory grows until the process is killed
-> restart nightly as an explicit stopgap,
with an owner and a date
-> profile the allocation growth
-> find and fix the retained reference
-> remove the scheduled restart
-> memory profile is flat; the class of
incident is gone rather than deferredBoth remove the immediate pain and only one removes the defect. The distinction that matters operationally is that the automated version makes the defect cheaper to keep, so it is kept — and defects that are kept tend to get worse, at which point the automation that was hiding them becomes the thing that breaks. Note that the better column still starts with the stopgap. The difference is that it has an owner and an expiry rather than becoming the answer.
What to do with toil once you can see it
Measurement is worth almost nothing on its own; what makes it useful is having named responses, so a rising number produces a decision instead of a discussion.
A recurring operational task is consuming a growing share of the team's week.
when The task exists because of a defect, a missing limit, or a design that requires attention to keep running.
cost Real engineering time, competing with feature work, on a fix with no visible customer.
when The task is legitimate, understood and standardised, and will recur regardless (How to Automate Something).
cost An automation to own, monitor and keep current as the system changes.
when The toil is other teams asking you to do things you could let them do safely (Self-Service Infrastructure).
cost Building and maintaining the interface, which is more work than doing the requests for a while.
when Many teams have the same toil, and the fix belongs once rather than per team (Platform Engineering).
cost Only viable if a platform exists; otherwise this is proposing to build one.
when A service generates disproportionate toil because it was never made operable (Production Readiness Review).
cost An organisational conversation, and it will be about the service's owner rather than about the work.
when It is small, does not scale with growth, and automating it costs more than it saves.
cost Nothing, provided the decision is explicit and revisited when the multiplier changes.
How to do it properly
Most important first.
- Measure it before arguing about it. A simple record of interrupt-driven work per person per week — what, how long, which service — is enough to change the conversation.
- Apply the six-property test explicitly, because the interesting outcome is usually finding out the work is *not* toil and needs a different response.
- For each recurring item, ask first whether the cause can be removed. Automating around a defect is the second-best answer and often gets chosen because it is faster (How to Automate Something).
- Cap toil deliberately as a fraction of a team's time, and treat exceeding the cap as a signal that engineering time must be reallocated, not that people should work more.
- Attribute toil to the service that generates it. Toil concentrated in one service is a readiness problem with that service, not a team capacity problem (Production Readiness Review).
- Route toil-reduction work through the normal engineering backlog with normal prioritisation, so it competes visibly rather than happening when someone is annoyed enough.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Unmanaged toil rarely causes a single event. It surfaces as a team with no engineering capacity and an on-call rotation people leave, which is contained only by measuring the load and reallocating time deliberately.
What can go wrong
- Toil that is invisible because it is absorbed by the most senior person, who is fastest at it and therefore complains least.
- Automating the symptom — a nightly restart, a scheduled cache clear, a script that reruns the failed job — which makes the underlying defect survivable and therefore permanent.
- Toil reclassified as "on-call duty" so it never appears in planning.
- A toil budget treated as a target to report rather than a signal to act on, so the number is managed instead of the work.
- Automation that reduces toil for the team that built it and creates toil for the team that has to operate it.
- "All manual work is toil." Manual work that does not recur, or that produces lasting change, is just work. Applying the label to everything makes it useless as a prioritisation tool.
- "Toil should be zero." Some operational work is irreducible and some automation costs more than the toil it removes. The target is that it does not grow with the system.
- "A toil budget is a rule." The commonly cited cap of roughly half an engineer's time is an organisational convention, not a law of nature — the useful part is having any explicit threshold that triggers a reallocation decision.
- "We automated it, so the toil is gone." The toil is smaller. Now there is an automation to own, monitor and keep current (How to Automate Something).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can state roughly how much of the last month went to interrupt-driven operational work, per person, and which services generated it.
- That fraction is trending down, or the growth in services explains why it is not.
- At least one recurring task was eliminated at the source in the last quarter rather than scripted around.
- New services do not add a fixed weekly operational cost per service.
- Toil reduction is normally an engineering change to a system, and rolls back like any other change.
- The reduction that does not roll back cleanly is deleting the manual procedure. Keep the runbook after automating it, because the automation will one day be the thing that is broken (Runbooks).
- Automate toil that is understood, standardised and genuinely repetitive — that is the definition's promise.
- Do not automate work that only looks like toil: incident response, capacity judgement, and decisions that vary with context are repetitive in *shape* and not in *content*.
- Automate the measurement of toil before the toil itself. A team that cannot see its operational load cannot prioritise reducing it.
- Eliminating the cause is usually more expensive than automating the task, and it competes with feature work that has a visible customer.
- Automation is not free of toil; it moves it to maintaining the automation, which is smaller and does not vanish.
- Measuring toil adds a small amount of overhead to the people already carrying it, which is a genuine and slightly ironic cost.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICThe toil cap — the widely quoted version is around half of an engineer's time — is a convention from published site reliability practice, not a measurement or a rule. What matters is that some explicit threshold exists and that crossing it triggers a decision about where engineering time goes; the specific fraction should be set against your own team size and service count.
- SCALE-SPECIFICAt small scale, toil that scales linearly is invisible because the multiplier is small — two services generating an hour a week each is unnoticeable. The property that makes it worth a name only bites past the point where per-service operational cost times service count exceeds what the team can absorb, which is usually reached suddenly.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how much recurring operational work is really a gap in what is verified before release.