DebuggingGENERALSCALE-SPECIFICORG-SPECIFIC
Production Anti-Patterns
The practices that reliably produce incidents — each with why it is tempting, because a list that only says "do not" teaches nothing.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
Which habits produce most production incidents, and why do sensible teams adopt them anyway?
The problem
Every one of these is adopted by competent people for a real reason, usually under time pressure. They are not ignorance; they are local optimisations whose cost lands later, elsewhere, and on someone else.
What teams do first
Publish the list of things not to do and expect the list to change behaviour. Everyone nods, and then ships a hotfix over SSH on Friday because the pipeline takes forty minutes.
How it breaks
A prohibition without an alternative loses to the pressure that produced the behaviour. If the safe path is slower than the unsafe one at 2am, the unsafe one wins.
How it breaks in production
- A prohibition without an alternative loses to the pressure that produced the behaviour. If the safe path is slower than the unsafe one at 2am, the unsafe one wins.
- People who do not know why a practice is tempting cannot recognise themselves doing it — they think the anti-pattern is something other, less careful teams do.
- Each of these has a legitimate emergency form. Banning it outright means it happens anyway, undocumented, which is strictly worse than a sanctioned break-glass path (Break-Glass Access).
- Lists get adopted as compliance checklists, so teams optimise the appearance rather than the property — access exists but is unreviewed, runbooks exist but are stale.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Nearly all of these share a shape: they trade a known, deferred cost for an immediate saving, and the deferred cost is paid during an incident by whoever is on call.
- The second shared shape is destroying evidence or reversibility. Manual patches, per-environment rebuilds and mutable tags all leave you unable to answer "what is running and where did it come from" at the moment you most need to.
- The third is removing a signal: alerting on everything removes the meaning of an alert, and ignoring failed backups removes the only evidence that recovery is possible.
- The fix is almost never willpower. It is making the safe path the fast path, so the tempting shortcut stops being the shortest route (Golden Paths).
Change and access
This group shares one property: the state of production stops being derivable from anything you can read. Once that is true, every subsequent incident starts with archaeology.
| Anti-pattern | Why it is tempting | What it actually costs | Instead |
|---|---|---|---|
| SSH in and manually patch production | It is the fastest path to a fix at 2am, and the pipeline takes forty minutes | The running state now matches no artifact. It vanishes at the next deploy or restart, and the next person cannot reproduce it | Make the emergency path a fast, audited deploy; treat the pipeline's speed as the actual bug (Manual Production Changes) |
| Unlimited standing production access | Restricting it slows people down, and everyone here is trusted | One compromised laptop or one mistyped command has unlimited blast radius, and nothing distinguishes a routine action from an unusual one | Read access broadly, write access scoped and time-bound, with break-glass for the rest (Least Privilege in Production) |
| Manual repetitive release steps | Each step is small, and automating them is a project nobody has time for | Steps get skipped or reordered under pressure, and the release becomes dependent on who runs it | Automate the sequence; keep the go/no-go decision human (Toil) |
| Configuration known only by one person | It works, they know it, and writing it down never becomes urgent | That person becomes a single point of failure for every incident, including the ones during their holiday | Configuration in version control, reviewed and applied through a pipeline (A Config Change Is a Production Change) |
| Huge, infrequent releases | Releasing is painful, so it is done rarely — which is a locally rational response to pain | Every release contains dozens of changes, so correlation is useless and rollback reverts everything, including the parts that worked | Smaller, more frequent releases; the pain is the signal to fix the path (Change Size: Why Small Changes Are Safer, and When They Are Not) |
Build, deploy and rollback
GENERALThese hold on any platform that has artifacts and releases. On a platform where deployment is a git push and the provider builds for you, "build once" is partly the provider's responsibility — the question becomes whether it rebuilds on promotion, which many do.
This group destroys either the identity of what is running or the ability to get back. Both are cheap to preserve at build time and impossible to reconstruct during an incident.
The migration item is the highest-severity entry on the whole list, because it is the one where "roll back" stops being available at all.
| Anti-pattern | Why it is tempting | What it actually costs | Instead |
|---|---|---|---|
| Rebuild separately for each environment | It feels natural — each environment gets a fresh build with its own settings | What you tested is not what you shipped. Any difference in inputs between builds is invisible and untestable | Build once, promote the same artifact through environments, vary only config (Build Once, Deploy Many) |
Deploy a mutable tag such as latest | It is convenient, always current, and requires no version bookkeeping | You cannot say what is running, cannot reproduce it, and a rollback may retrieve different bits than it did yesterday | Deploy by immutable digest; keep tags as human labels only (Tags Versus Digests) |
| No rollback plan | Forward-only feels confident, and rollback is assumed to be automatic | The first time you need it is during an incident, which is where you discover it was never tested | Decide and test the reversal before the change ships (Rollback: Only Useful If It Is Actually Safe) |
| Destructive DB migration in the same release as the code | One deploy, one change, one thing to coordinate — simpler on paper | The code can roll back and the dropped column cannot. The release becomes one-way at the moment you most want it reversible | Expand, migrate, contract as separate releases (Expand, Migrate, Contract) |
| Secrets committed to git | It is expedient during setup and everything works immediately | They live in history forever, on every clone and every fork, and rotation is now a separate project | Secret manager or workload identity from the start; scan history and rotate what is there (What Counts as a Secret, and Where It Must Not Be) |
| CI runners with broad standing production credentials | It makes every pipeline "just work" without per-job permission wrangling | Any dependency in any build now has production credentials — the widest supply-chain surface most teams have (Securing the Pipeline Itself) | Short-lived, job-scoped credentials issued per deployment (Secrets in CI) |
Signals, recovery and scale
SCALE-SPECIFICThe orchestration and autoscaling entries are about fit, not about the tools. Kubernetes is the correct answer for many organisations; the anti-pattern is adopting it as a definition of production rather than as an answer to placement, rollout and bin-packing problems you can name (The Problems Kubernetes Answers).
This group removes your ability to know something is wrong, to recover when it is, or to understand the system you have chosen. They are the quietest items on the list and the most common contributing factors in severe incidents.
| Anti-pattern | Why it is tempting | What it actually costs | Instead |
|---|---|---|---|
| Alert on everything | Every alert was added after a real incident, by someone being responsible | Volume destroys meaning. Real pages are missed inside noise, and on-call burns out (Alert Fatigue) | Page on user-visible symptoms; everything else is a dashboard or a ticket (Alert on Symptoms, Not on Causes) |
| No runbooks | The people who know the system are on call, so writing it down feels redundant | The knowledge is unavailable exactly when the person holding it is asleep, on leave, or has left | A runbook per alert, written for someone who was not there (Runbooks) |
| Ignore failed backup jobs | The failure is usually transient, the job is noisy, and it has always been fine | Backups are only evidence if they succeeded. A silently failing job means the last good copy is older than anyone believes | Alert on backup failure and on backup age; treat both as user-facing (Backup Operations) |
| No restore drills | Backups exist and are green, so restoring is assumed to work | You discover the restore path is broken, incomplete or far slower than your recovery objective during the disaster | Scheduled restore drills with timing, into a real environment (Restore Drills) |
| Kubernetes because everyone uses it | It is the default answer, it is well-documented, and it is good for hiring | A large operational surface with its own failure modes, adopted before the problems it solves exist (Do You Need Kubernetes?) | Choose the simplest platform that solves the problems you actually have, and revisit as they change |
| Autoscaling without understanding capacity | It appears to make capacity someone else's problem | Scaling on the wrong signal, or into a downstream bottleneck, converts a slow service into a saturated database (How Autoscaling Fails) | Model capacity first — what saturates, at what level, with what lag — then automate within it (Building a Capacity Model) |
How to do it properly
Most important first.
- For each anti-pattern present in your system, identify the pressure that produces it and remove the pressure, not just the behaviour. A slow pipeline creates SSH patching; speeding the pipeline removes it (CI Is a Feedback System).
- Provide a sanctioned, audited emergency path for the cases that genuinely need one, so the emergency does not have to route around the controls (Break-Glass Access).
- Prioritise by blast radius rather than by list order. Unlimited production access, destructive migrations and untested restores are categorically worse than a stale runbook.
- Convert conventions into mechanisms: immutability enforced by the registry beats a rule about tags, and least privilege enforced by policy beats a norm about care (Policy as Code).
- Review the list against your own last five incidents, not in the abstract. The ones you have already been bitten by are the ones worth fixing first.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
These are the practices that specifically remove containment — rollback, audit, least privilege and recovery. That is why they cluster in postmortems.
What can go wrong
Failure modes, including of the mitigation
- Fixing the behaviour while leaving the pressure, so it reappears in a new form — SSH access removed, and now people push emergency commits straight to the release branch.
- Adopting all the controls at once on a small team, producing so much friction that people build a shadow path around them.
- Treating the list as a maturity score, so effort goes to items that are cheap to demonstrate rather than items that are dangerous.
- Guardrails with no exception path, which convert every genuine emergency into a policy violation performed in silence.
Misreads this invites
- "These are things bad teams do." Every one of them is a rational local decision under pressure. The good teams differ in having removed the pressure.
- "Never SSH into production." Investigating from inside a running container is sometimes exactly right (Debugging a Container in Production). The anti-pattern is *changing* production that way, leaving state no artifact reproduces.
- "Adopt everything on this list." Several items are scale-dependent, and adopting them all on a small system produces ceremony without safety (What an Environment Is For).
- "We have controls, so we are fine." Controls that are routinely bypassed are worse than absent ones, because they also produce false confidence.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- You can name which of these are true of your system today, with an example of each, rather than asserting none apply.
- The emergency path exists, has been used, and produced an audit record and a follow-up (Access Review).
- The last incident postmortem did not name one of these as a contributing factor twice in a row.
How you get back
- Removing an anti-pattern is itself a change with a blast radius: tightening production access can lock out the person who needs it at 3am. Stage it, and keep a break-glass route open while you do.
- Where a control turns out to be too tight, loosen it deliberately and record why, rather than letting it be bypassed informally.
What to automate, and what stays human
- Automate detection: unpinned tags, publicly writable buckets, secrets in git history, standing production credentials in CI, backup jobs that failed silently.
- Automate the safe path so it is faster than the unsafe one — that is the only intervention that reliably works (Golden Paths).
- Keep the exception human and reviewed. An automated emergency-access grant with nobody approving it is simply standing access with extra steps.
What this costs
- Every control here costs friction, and friction is not free — it slows recovery as well as mistakes. The right amount depends on blast radius and team size.
- Some items are genuinely scale-dependent: a two-person team with one service does not need environment promotion machinery, and adding it costs more than it returns.
- Auditing and least privilege add real latency to legitimate emergency work. The answer is a fast sanctioned path, not the absence of controls.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe failure mechanism of each holds broadly. What varies is severity: mutable tags on a hobby project cost an afternoon, and on a regulated system cost your ability to say what was running during an audited period.
- SCALE-SPECIFICSeveral items only become anti-patterns above a threshold. One person deploying one service does not need environment promotion or formal access review; both become necessary as soon as more than one person can change production and nobody sees all the changes.
- ORG-SPECIFICWhere the line sits between guardrail and gate is an organisational choice. A team with strong review culture may allow broader standing access than one where changes are made by contractors on rotation; neither is universally correct.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Backenddeployment-models
Observability & Performancealert-fatigue