Kubernetes Anti-Patterns
The recurring mistakes that produce most cluster incidents — each one reasonable at the moment it is made, and each one with a specific failure it eventually causes.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Which cluster practices reliably produce incidents, and what does each one actually break?
Kubernetes accepts almost anything you declare. Most of these mistakes have no immediate symptom, which means the feedback arrives weeks later as an incident with no obvious connection to the decision that caused it.
Assume that if the manifest applies, the pods run and traffic is served, the configuration is fine. The platform validates what it can, so what it accepts must be reasonable.
Validation checks shape, not judgement. A pod with no resource requests, a liveness probe that calls a database and an image tagged latest are all perfectly valid.
- Validation checks shape, not judgement. A pod with no resource requests, a liveness probe that calls a database and an image tagged
latestare all perfectly valid. - The consequences are load-dependent and failure-dependent. Everything works until a node is under pressure, a dependency is slow, or a rollout happens at a bad moment.
- By the time the incident arrives, the manifest has been copied into a dozen other services, so the fix is a fleet-wide change rather than a one-line edit.
- Cluster mistakes tend to look like application failures — restarts, latency, unavailability — so the investigation starts in the wrong place (Reading a Broken Workload).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The pattern behind almost all of them is the same: a field that is optional, whose default is "unbounded" or "as often as possible", left unset because nothing complains.
- No requests means the scheduler places blind. It cannot reserve what you did not declare, so nodes get packed until something is evicted or throttled (Requests and Limits).
- Liveness that depends on the world converts a dependency outage into a restart storm: every replica fails its probe simultaneously, is killed, and restarts into the same failing dependency (Probes: Readiness, Liveness and Startup).
- A mutable tag breaks the link between what you tested and what is running. Two pods created from the same manifest an hour apart can be running different code (Tags Versus Digests).
- Manual changes create state that exists in the cluster and in nobody's repository. It survives until the next apply, then disappears without a trace (Manual Production Changes).
- Excessive service granularity multiplies the network hops, failure modes, deploy surfaces and dependency edges on every request path, and the coordination cost grows faster than the number of services (Microservices).
Seven mistakes and what each one actually causes
Each row is defensible at the moment it is made. Read the cause column as the delay between the decision and the consequence — that delay is why these persist.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| No resource requests or limits | Random evictions and unexplained latency under node pressure | The scheduler places against requests; with none it packs blind and the kernel arbitrates instead | Set memory requests and limits and CPU requests; enforce at admission (Requests and Limits) |
| Liveness probe calls a dependency | A dependency blip becomes a fleet-wide restart storm | Every replica fails simultaneously and is killed, then restarts into the same failure | Liveness tests the process; readiness tests the ability to serve (Probes: Readiness, Liveness and Startup) |
latest or a moving tag | Two pods from one manifest run different code; a rollback delivers the wrong version | The tag is a pointer, and it moved between pulls | Deploy by digest; treat tags as human labels (Tags Versus Digests) |
| Too many tiny services | One request crosses eight hops; every incident is a distributed trace | Failure modes and coordination cost grew faster than the service count | Merge what shares an owner and a release cadence (Modular Monolith) |
| Careless secret handling | Credentials in logs, in crash dumps, in git history | Base64 read as protection; environment variables read as private | Mount as files, scope RBAC per name, use a secret manager (ConfigMaps and Secrets) |
| Manual cluster changes | A fix that works and vanishes at the next apply; state nobody can reproduce | Live state diverged from the repository with no record | Apply from a reviewed repository; make direct edits break-glass (Manual Production Changes) |
| Stateful systems without a persistence model | Data loss on a routine node drain | emptyDir or an unowned volume treated as durable storage | Decide the lifetime explicitly; prefer a managed service (Volumes: Storage With a Lifecycle, Why Stateful Workloads Are Harder) |
The probe mistake, in detail
This one deserves its own treatment because it is the most damaging and the most intuitive-seeming. It is entirely reasonable to think that a service which cannot reach its database is unhealthy and should be restarted.
It is reasonable and it is wrong, because restarting does not fix a dependency and because all replicas fail the check at the same instant. A degraded service that returns errors is strictly better than no service at all — and it is still there when the dependency recovers.
`/health` verifies the database connection, the cache and a downstream API, and is wired to the liveness probe. When any of them is slow, the probe times out.
`/live` returns 200 if the process is running and not deadlocked, and nothing else. `/ready` checks whether this replica can serve — including dependencies it truly cannot function without — and is wired to readiness only.
Liveness answers "should this process be killed", and the only correct reason is that restarting will help. Readiness answers "should traffic come here", and removing a replica from rotation is reversible while killing it is not. With the worse form, a five-second database blip kills every replica at once and the restarts then hammer the recovering database (Retry Storms: The Load You Generated Yourself).
What to enforce, and where
Enforcement point matters more than the rule. A convention in a wiki is followed by whoever read the wiki; a rejection at admission is followed by everyone, immediately, with feedback attached to the change that caused it.
The reason to draw this table is that the enforcement point determines the cost of the mistake, not just whether it is caught.
| Rule | Best enforcement point | What it costs when unenforced |
|---|---|---|
| Resource requests present | Admission policy, rejecting the manifest | Evictions and throttling under load, attributed to the application |
| Image referenced by digest | Pipeline, injecting the digest | Unreproducible deploys and unreliable rollback |
| Liveness independent of dependencies | Review, plus a template default | Restart storms during every dependency incident |
| No direct cluster edits | RBAC plus drift detection | Live state nobody can reproduce or explain |
| Secrets not in environment variables | Template default, plus a manifest scan | Credentials in crash dumps and error reports |
| Persistence model chosen deliberately | Review at service creation | Data loss during a routine node drain |
How to do it properly
Most important first.
- Set memory requests and limits on every container, and set CPU requests. Be deliberate about CPU limits — they throttle rather than kill, and a limit set too low is a self-inflicted latency problem (CPU Throttling: The Latency With No Error).
- Make liveness test the process and readiness test the ability to serve. Liveness must not depend on any downstream system (Probes: Readiness, Liveness and Startup).
- Deploy by digest. Tags are for humans; digests are what the cluster should run (Tags Versus Digests).
- Apply from a reviewed repository and make direct cluster edits a break-glass action with an audit trail (Break-Glass Access).
- Enforce the non-negotiable parts with admission policy, so a manifest missing them is rejected rather than discovered in an incident (Policy as Code).
- Before adding a service, ask what it would take to make it a module in an existing one. The default should be the smaller number of deployables (Modular Monolith).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Several of these fail cluster-wide by design — a restart storm, an evicting node, a moved tag — and the containment is admission-time rejection rather than anything at runtime.
What can go wrong
- Node pressure evicting the wrong pods, because the pods with no requests are indistinguishable from the ones that were carefully sized.
- A dependency blip becoming a full outage as every replica restarts at once, and the restart storm keeping the dependency down.
- A rollback to a tag that has since moved, delivering something other than the version you rolled back to.
- A cluster whose live state cannot be reproduced from the repository, so a rebuild after a serious failure is archaeology (Drift).
- A change to shared policy — resource defaults, network policy, admission rules — that is correct for one team and breaks three others.
- Secrets handled casually: read access granted broadly, values in environment variables, base64 committed to the repository because it did not look like a password (ConfigMaps and Secrets).
- "It works, so the configuration is fine." Most of these have no symptom until load, a failure or a rollout arrives.
- "Limits everywhere is always right." A CPU limit below what the workload needs throttles it into latency nobody can explain; the mistake is applying a rule without measuring (CPU Throttling: The Latency With No Error).
- "More services is more scalable." More services is more network hops, more failure modes and more coordination. Scalability is a property of the design, not the count (Microservices).
- "We will add probes and limits later." Later is after the manifest has been copied into every other service.
- "Anti-patterns are a Kubernetes problem." Every one of these has an equivalent elsewhere: unbounded processes on a VM, health checks that test the world, floating version tags, hand-edited servers.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Admission policy rejects a manifest with no resource requests, and someone has seen it do so.
- A deliberately induced dependency outage produces degraded responses rather than a restart storm.
- Every running pod's image is a digest, verifiable from the cluster in one query.
- The cluster's live state matches the repository, checked continuously rather than assumed (Drift).
- Individual fixes roll back like any other manifest change, and the risky ones are the fleet-wide corrections: adding limits everywhere at once can start evicting workloads that were quietly over-consuming.
- Roll these out the way you would any other risky change — one namespace, one workload class at a time, watching for eviction and throttling (Change Size: Why Small Changes Are Safer, and When They Are Not).
- The one that does not roll back is a secret that has been exposed. That rotates (Rotation That Applications Survive).
- Automate enforcement at admission, which is the only point where the feedback is immediate and unambiguous (Policy as Code).
- Automate drift detection between repository and cluster, since manual changes are invisible by construction (Drift).
- Automate resource recommendations from observed usage, and keep the decision to apply them human — a recommendation from a quiet week is a bad limit (Building a Capacity Model).
- Admission policy prevents real incidents and makes the platform stricter than the tutorials, which is friction that has to be explained to every new team.
- Requests and limits everywhere improves scheduling and requires ongoing tuning, since a limit set once is wrong after the workload changes.
- Fewer, larger services reduce operational surface and give up independent deployment for the parts that genuinely needed it (Microservices).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe mechanisms — evictions, probe-driven restarts, admission policy — are Kubernetes'. The mistakes are not: a VM fleet has unbounded processes competing for memory, health checks that test downstream systems, and floating version tags, with the same consequences and worse visibility.
- SCALE-SPECIFICMissing requests and limits are harmless on a lightly loaded cluster and become the dominant failure mode as node utilisation rises. Service granularity has the opposite shape: five services is fine at any scale, and fifty is only manageable with real platform investment.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.