The question this answers
What is the difference between "restart this process" and "stop sending it traffic", and why does confusing them cause outages?
A service is temporarily unable to serve — its cache is cold after startup, or its database is failing over. It should stop receiving requests. It should absolutely not be killed, because killing it makes the situation worse and the process was never the problem.
Two independent decisions with independent consequences: traffic removal for a temporary inability to serve, and a restart for a process that is genuinely stuck.
Two questions, two consequences
Liveness: is this process alive enough to continue existing? A failed liveness probe means "this process cannot recover on its own — kill it and start a new one". The correct answers are deadlock, an event loop that stopped turning, a state machine wedged beyond repair. The remedy is destructive and only justified when a restart is genuinely the fix.
Readiness: can this process serve traffic right now? A failed readiness probe means "route around this instance for the moment". It is non-destructive and completely reversible. Correct answers here are: still warming up, connection pool momentarily empty, draining before shutdown, deliberately shedding load.
The distinction is not academic pedantry about probe names. It is the difference between a reversible action and an irreversible one. Readiness failing is a routing change; liveness failing destroys the process, its warm caches, its open connections and any in-flight work. Attach the wrong probe to the wrong condition and the platform will helpfully apply the destructive remedy to a problem restarts cannot solve.
| Condition | Liveness should fail? | Readiness should fail? | Why |
|---|---|---|---|
| Process deadlocked, event loop stuck | Yes | Yes (it follows) | A restart is the only available fix |
| Still warming up: pools opening, cache filling | No | Yes | Restarting resets the warm-up you were waiting for |
| Shared database failing over | No | Debatable — usually no | Restarting every pod cannot bring the database back and destroys the fleet meanwhile |
| Connection pool momentarily exhausted | No | Yes | Self-correcting in seconds; a restart makes it much worse |
| Shutting down, draining in-flight requests | No | Yes | Stop new traffic, finish what you have (Graceful Shutdown: The 502 Spike Nobody Investigates) |
| Unrecoverable internal state, corrupted in-memory data | Yes | Yes | The process cannot be trusted to serve anything |
| Deliberately shedding load under pressure | No | Yes | A load-shedding signal, not a defect |
The outage: a liveness probe that checks the database
The single most expensive version of this mistake is one line of configuration. Someone points the liveness probe at /health, and /health checks the database. It works perfectly for a year.
Then the database fails over for twenty seconds. Every pod fails its liveness probe. The orchestrator does exactly what it was told and kills every pod. New pods start, find the database still unavailable during their startup, fail liveness again, and are killed again — now with backoff. The database returns after twenty seconds, and the fleet does not, because it is in CrashLoopBackOff with an exponentially increasing delay, every cache is cold, every connection pool is closed, and the stampede of restarting pods reconnecting all at once knocks the recovering database over a second time.
A twenty-second dependency blip has become a twenty-minute outage, and the cause is that a *degradation* was expressed as a *crash*. Readiness on the same condition would have removed the pods from service for twenty seconds and put them straight back, with warm caches and open pools, and most users would never have noticed. The rule that follows: a liveness probe must never depend on anything outside the process. If a restart cannot fix it, liveness must not test it.
- 1t+0s — failover begins
The database primary steps down. Connections are refused for about twenty seconds.
Entirely survivable. The application could return cached data or degrade gracefully.
- 2t+10s — liveness fails everywhere
Each pod's
/healthqueries the database and times out. Failures accumulate past the threshold simultaneously.Perfectly correlated across the fleet, because the dependency is shared.
- 3t+15s — the orchestrator kills every pod
Containers are killed and restarted. Warm caches, open pools and in-flight requests are destroyed.
The destructive remedy has now been applied to a problem restarts cannot address.
- 4t+25s — database recovers, fleet does not
New pods are starting cold. Startup takes longer than the failover did.
The outage now outlives its cause.
- 5t+40s — CrashLoopBackOff
Repeated failures push restarts into exponential backoff: 10s, 20s, 40s, 80s.
The platform is now actively delaying recovery, correctly following the instruction it was given.
- 6t+2m — reconnection stampede
Backoff expires in waves; every pod opens its full connection pool against the freshly recovered database at once.
The database is overwhelmed a second time, this time by you.
- 7t+20m — manual recovery
An engineer edits the probe, or scales the deployment down and back up in stages.
The post-mortem blames the database failover, and the probe survives to do it again.
What the configuration should say
Written down, the correct version is barely longer than the wrong one. Liveness checks a trivial in-process signal on a generous interval with a tolerant threshold — it is a last resort, not a monitor. Readiness checks what this instance needs to serve, on a short interval, with a tight threshold, because it is reversible and cheap to be wrong about. A startup probe (or an equivalent initial delay) covers slow initialization so that liveness does not start killing a process that is merely still booting.
A useful discipline: write down, for each thing a probe checks, the sentence "if this fails, restarting the process will fix it." If the sentence is false, the check belongs to readiness or to monitoring, never to liveness.
And keep the endpoints separate. One path per probe, doing exactly what its probe means. Reusing a single /health for both is how the two decisions get quietly merged again six months later, by someone who adds a dependency check to the endpoint without knowing what else consumes it.
livenessProbe:
httpGet: { path: /health, port: 8080 }
periodSeconds: 5
failureThreshold: 2 # kills after ~10s of failure
readinessProbe:
httpGet: { path: /health, port: 8080 }
periodSeconds: 5
# /health does: SELECT 1 against the primary database.
# A 20s failover kills 100% of pods within 10 seconds.startupProbe: # covers slow init; liveness waits for it
httpGet: { path: /livez, port: 8080 }
periodSeconds: 5
failureThreshold: 30 # tolerate up to ~150s of startup
livenessProbe:
httpGet: { path: /livez, port: 8080 } # in-process only:
periodSeconds: 15 # event loop turning,
timeoutSeconds: 3 # no deadlock detected
failureThreshold: 4 # ~60s of sustained failure before kill
readinessProbe:
httpGet: { path: /readyz, port: 8080 } # this pod can serve:
periodSeconds: 5 # warm-up done, pool open,
timeoutSeconds: 2 # not draining
failureThreshold: 2Liveness now tests only facts a restart could plausibly fix, on a tolerant schedule, so a shared dependency can never trigger a fleet-wide kill. Readiness reacts fast because removing traffic is reversible. The startup probe stops liveness from executing a process that is simply still booting.
Key points
- Liveness answers "should this process be killed?"; readiness answers "should this instance receive traffic?". The consequences are destructive and reversible respectively.
- A liveness probe must never test anything outside the process. If a restart cannot fix it, liveness must not check it.
- A liveness probe that checks a shared database restarts the entire fleet when the database blips, converting a degradation into a crash loop.
- Readiness may fail often and briefly — during warm-up, during draining, under load shedding. That is the mechanism working correctly.
- Use a startup probe or a generous initial delay so slow initialization is never mistaken for a wedged process.
- Give each probe its own endpoint. A shared
/healthmerges the two decisions back together the moment someone adds a dependency check.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The node agent (or platform equivalent) executes each probe against the container on its own schedule.
- • Readiness failure removes the instance from the service endpoint list, so traffic stops arriving. State is untouched and return is immediate once it passes again.
- • Liveness failure past its threshold causes the container to be killed and restarted, subject to the restart policy and an exponential backoff.
- • A startup probe, while it is running, suspends liveness and readiness entirely — which is what prevents a slow boot from looking like a deadlock.
- • Repeated liveness failures push restarts into increasing backoff, which is the platform protecting itself and, in the failure above, actively delaying your recovery.
- • Audit every liveness probe for external dependencies. This is a fifteen-minute exercise that prevents a specific, expensive outage.
- • Set liveness thresholds tolerantly: it is the last resort, and a false positive costs a process. Set readiness thresholds tightly: a false positive costs a few seconds of one instance's traffic.
- • Make readiness aware of shutdown: fail readiness first, wait for the endpoint list to converge, then stop accepting connections (Graceful Shutdown: The 502 Spike Nobody Investigates).
- • Watch restart counts per workload as a first-class signal. A slowly rising restart count is a probe misconfiguration or a leak, and it is visible long before it becomes an incident.
- • Load-test with the probes enabled. A readiness probe that times out under load will remove your capacity at exactly the wrong moment.
- • Liveness checking a shared dependency: fleet-wide crash loop from a transient blip, with backoff extending the outage past its cause.
- • Liveness too aggressive: a slow startup or a garbage-collection pause is read as a deadlock and the process is killed mid-warm-up.
- • No readiness probe: traffic arrives at a process that has not finished initializing, and every deploy produces a burst of errors.
- • Readiness that never fails: broken instances stay in rotation, and the mechanism silently provides nothing.
- • Both probes on one endpoint: a colleague adds a dependency check for good reasons and unknowingly arms the crash loop.
- • Readiness flapping under load: instances oscillate in and out, traffic concentrates on whoever is currently in, and the oscillation spreads across the fleet.
- • Probe execution cost is per-instance and constant, but the load it places on anything readiness touches scales with the fleet.
- • Correlated probe failures are the dangerous class: anything shared — a database, a config service, a token endpoint — can take the whole fleet at once if it appears in a probe.
- • Endpoint-list propagation is not instant at large scale; a readiness change on 500 instances takes measurable time to reach every proxy.
- • Larger fleets need more tolerant readiness thresholds, because the probability that some instance is momentarily busy approaches one.
- • Probe endpoints are called by the node agent and must not require authentication, which makes them reachable from inside the network by default. They must disclose nothing.
- •
/readyzreturning a list of failing dependency hostnames is an internal reconnaissance gift. Return a status, not a diagnosis. - • A liveness endpoint that performs real work is a denial-of-service amplifier for anyone who can reach it — keep it trivial.
- • Restart loops erase container filesystems and in-memory state. If anything security-relevant lives only there — audit buffers, rate-limit counters — it is destroyed by every liveness kill (Audit Trails).
- • Probes themselves are negligible. Crash loops are not: every restart repays the full image pull, boot and warm-up cost.
- • A misconfigured liveness probe across a large fleet generates continuous restart churn — registry pulls, cold caches, database reconnections — that shows up on several meters at once.
- • Readiness done well saves money indirectly: fewer failed requests means fewer client retries, and retries are load you pay to serve twice.
- • Time is the real cost. The outage in this lesson is twenty minutes of a service being down because of one YAML path.
- • Container restart count per workload, alerted on rate of change. This is the highest-signal, lowest-effort probe metric there is.
- • Ready-replica count against desired replica count — the gap is exactly the traffic-serving capacity you have lost.
- • Probe failure events with their reasons, which distinguish "timed out" from "returned 503" and therefore distinguish overload from dependency failure.
- • Time spent not-ready per instance after a deploy, which is your real rollout duration.
- • The signal that lies: pods reported as
Running. A pod in a crash loop spends most of its timeRunningbetween kills, and a pod that is Running but not Ready serves nothing.
- • Readiness alone. Many services genuinely do not need a liveness probe: if the process cannot get stuck in a way a restart fixes, an absent liveness probe is strictly safer than a badly written one.
- • External supervision only — a process manager restarting on exit, with the load balancer handling routing. On a small VM fleet this covers both concerns without any probe configuration.
- • Passive detection: eject instances based on real request error rates rather than a self-reported probe, which avoids inventing a predicate at all.
- • For a stateless process that exits on unrecoverable errors,
restart: alwaysplus a readiness check is the whole design. Not every workload needs three probes.
- • A tolerant liveness probe avoids killing healthy processes and leaves a genuinely wedged one running longer.
- • A fast readiness probe removes broken instances quickly and flaps under load, concentrating traffic on fewer instances.
- • A startup probe generous enough for the worst boot also delays detection of a process that is genuinely stuck at startup.
- • Separate endpoints are clearer and are two more paths to keep correct through refactors.
Liveness vs readiness, against the same database blip
What people believe, and what is true
Liveness and readiness can share one endpoint if it is written well.
They can, until someone adds a dependency check to it. Separate paths make the two meanings impossible to merge by accident.
A failing readiness probe is an incident.
It is the mechanism working. Instances are meant to be not-ready during warm-up, draining and load shedding.
Restarting a pod is cheap, so an aggressive liveness probe is safe.
A restart destroys warm caches and open pools, and repeated restarts hit exponential backoff. Applied fleet-wide it is an outage, not a mitigation.