ObservabilityGENERALCLOUD-SPECIFICRUNTIME-SPECIFIC

Health Checks: Startup, Readiness, Liveness

Three different questions with three different consequences — and a liveness check that fails on a dependency outage turns a bad hour into a much worse one.

What actually happensHow to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What should a health endpoint check, and what happens when it says no?

The requirement

The platform needs to know when a new instance can take traffic, when an instance should be taken out of rotation, and when one is wedged and should be restarted.

The obvious build

Expose GET /health that checks the database, the cache and the message broker, and return 200 only if all of them respond. Point the load balancer and the orchestrator at it. One endpoint, complete coverage.

Why it breaks

The database has a brief outage. Every instance fails its check simultaneously. The orchestrator, reading this as liveness, restarts every one of them — so when the database recovers, it is met by a fleet of cold processes with empty pools and full caches to rebuild.

How it breaks in production
  • The database has a brief outage. Every instance fails its check simultaneously. The orchestrator, reading this as liveness, restarts every one of them — so when the database recovers, it is met by a fleet of cold processes with empty pools and full caches to rebuild.
  • The restarts make it worse in a specific way: a restarting instance cannot serve the requests it *could* still have served, such as anything cached or any endpoint that does not touch the database.
  • A dependency that is slow rather than down causes the check to time out, and a health endpoint that queries the database under load adds load to the database that is already struggling.
  • The single endpoint cannot express "not ready yet" versus "permanently broken". At startup, before migrations and warm-up finish, it returns 503 — and a liveness probe reads that as a dead process and kills it before it ever starts, forever (Startup Time & Cold Start).
  • Nothing distinguishes "this instance is bad" from "everything is bad", so the load balancer removes all instances from rotation and returns 503 to every caller, when serving degraded responses from any of them would have been better.
RequirementAPI ContractApplication LogicData AccessExternal DepsConcurrencyFailureSecurityObservabilityDeploymentScale

What is actually happening

  • The three checks answer genuinely different questions, and the difference is entirely in what happens when the answer is no.
  • Startup — "has this process finished initialising?" Config loaded, migrations checked, connections opened, caches warmed. A failure means *keep waiting*, and a long grace period is correct. Its purpose is to stop a liveness probe from killing a slow-starting process.
  • Readiness — "should this instance receive traffic right now?" A failure means *remove from the load balancer*, no restart. It is reversible, and it should flip back the moment the condition clears.
  • Liveness — "is this process wedged beyond recovery?" A failure means *kill and restart*. This is the destructive one, and it must only test things a restart can fix.
  • The decisive rule follows directly: liveness must not depend on anything external. A restart cannot fix your database. Checking it in liveness converts a dependency outage into a restart loop, which is strictly worse than the outage alone (Cascading Failure).
  • What a restart *can* fix: a deadlocked thread pool, an exhausted heap, a blocked event loop, an unrecoverable internal state machine. Those are the legitimate contents of a liveness check.
  • Readiness can legitimately depend on a hard dependency — but only if degrading is worse than serving. If any part of your service works without the database, readiness should not fail on the database.
  • Both readiness and liveness are graded by the platform: consecutive failures, periods and timeouts decide how twitchy the system is. A single failed probe should rarely be decisive.

Three questions, three consequences

Work backwards from the consequence and the design becomes obvious. The question is never "what would be useful to check" — it is "what should happen if this check fails, and would that action help?"

Liveness is the only one that destroys something. That asymmetry is why it gets the shallowest check, which feels wrong to most engineers the first time they hear it.

CheckQuestionOn failureMay depend on a dependency?Typical content
StartupHas initialisation finished?Keep waiting; suppress liveness until it passesYes — you cannot serve until the pool existsConfig validated, migration version, pool open, cache warmed
ReadinessShould this instance get traffic?Remove from the load balancer. Reversible, no restartOnly what this instance truly needs to servePool healthy, not shutting down, not overloaded
LivenessIs this process unrecoverable?Kill and restart the processNo. Never.Handler executes; loop lag or thread-pool state within bounds
Diagnostics (not a probe)What is the state of everything?Nothing automatic — a human reads itYes, in full detailPer-dependency status, versions, pool stats — authenticated only

The restart loop, step by step

GENERALAny supervisor that restarts on a failed check has this shape. Kubernetes makes it most visible because livenessProbe is prominent and easy to point at a dependency, but a systemd watchdog or a cloud auto-healing group behaves identically.

This is worth walking through slowly because it is counter-intuitive: the failure is caused by the mechanism that exists to improve reliability, and every individual step behaves exactly as configured.

The critical detail is the second-order effect. Restarting does not merely fail to help — it destroys warm state, empties connection pools, and produces a thundering herd of reconnections at the exact moment the dependency is trying to recover (Cascading Failure).

check times outcorrelated, not independentrestart is the configured actionall pools reopen at onceload on a struggling dependencyDatabase briefly unavailableLiveness probe queries the DBEvery instance fails simultaneouslyOrchestrator kills every podCold starts: empty pools, cold cachesReconnection storm at the recovering DBRecovery now takes far longer than the fault
UserLLMAgentToolDataDecisionHumanGuardrail

What each endpoint contains

Concrete implementations settle most arguments. Note how little liveness does, how readiness reads a cached dependency status rather than probing, and how the shutdown flag is what makes graceful draining work.

Note also what is absent from every response: no version, no dependency names, no error text. The detail belongs on an authenticated diagnostics route.

Three endpoints, three semantics
1let startupComplete = false
2let shuttingDown = false
3
4// Refreshed on a timer, NOT per probe — probes must not load the dependency.
5const depStatus = { db: 'unknown' as 'ok' | 'degraded' | 'unknown', checkedAt: 0 }
6setInterval(async () => {
7 try { await pool.query('SELECT 1'); depStatus.db = 'ok' }
8 catch { depStatus.db = 'degraded' }
9 depStatus.checkedAt = Date.now()
10}, 5_000).unref()
11
12// LIVENESS — no I/O, no dependencies. Only things a restart can fix.
13app.get('/livez', (_req, res) => {
14 const lagMs = eventLoopLag() // runtime-specific wedge signal
15 res.status(lagMs < 5_000 ? 200 : 503).end()
16})
17
18// READINESS — should this instance receive traffic right now?
19app.get('/readyz', (_req, res) => {
20 if (!startupComplete || shuttingDown) return res.status(503).end()
21 // Only gate on the DB if nothing this service serves works without it.
22 if (depStatus.db === 'degraded' && !SERVES_CACHED_READS) return res.status(503).end()
23 res.status(200).end()
24})
25
26// STARTUP — one-time initialisation; a generous failureThreshold is correct.
27app.get('/startupz', (_req, res) => res.status(startupComplete ? 200 : 503).end())
28
29// Draining: readiness goes false FIRST, then in-flight work completes.
30process.on('SIGTERM', async () => {
31 shuttingDown = true // /readyz now 503; LB drains us
32 await sleep(DRAIN_GRACE_MS) // let the LB notice before we stop accepting
33 await server.close()
34 await logger.flush()
35 process.exit(0)
36})

The DRAIN_GRACE_MS sleep is the part most implementations omit. Between readiness turning false and the load balancer removing the instance there is a window measured in probe periods, and requests arriving in that window are dropped if the server closes immediately.

How to build it

Most important first.

  • Expose three separate endpoints, or one endpoint with three modes. Do not point liveness and readiness at the same logic (Liveness vs Readiness).
  • Liveness: return 200 if the process can execute code and its internal state is sane. No network calls. In practice this is close to "the handler ran".
  • Readiness: check only what this instance genuinely needs to serve *its* traffic, and prefer cached dependency status over live probing so the check adds no load to a struggling dependency.
  • Startup: check the things that must be true once — config validated, migrations at the expected version, pool established, essential caches warmed (Validate at Startup, Fail Loudly).
  • Flip readiness to false as the first action on receiving a shutdown signal, then drain in-flight work. This is what makes a rolling deploy lose zero requests (Graceful Shutdown).
  • Cache the results of any expensive check for a few seconds. Health endpoints get probed constantly, by every instance, and an uncached database ping per probe is real load.
  • Make health endpoints unauthenticated but uninformative: 200 or 503 and nothing else. A separate authenticated diagnostics endpoint can list per-dependency status for humans.
  • Grade the thresholds deliberately. Requiring three consecutive readiness failures before removal absorbs a single blip; a one-failure threshold makes the fleet oscillate.

What can go wrong

Failure modes
  • The restart loop: liveness depends on a dependency, the dependency degrades, every instance restarts repeatedly, and the recovery is much slower than the original fault. This is the single most damaging health-check mistake there is.
  • A shallow liveness check that always passes because it returns a constant — technically correct, but it will never catch the deadlocked pool it was meant to catch.
  • Readiness that never goes false, so a genuinely broken instance keeps receiving traffic and the load balancer keeps sending it there.
  • Readiness flapping: an instance oscillating in and out of rotation, redistributing load each time and causing the very saturation that triggered the flap.
  • The health check itself blocking the runtime — a synchronous database ping on Node's loop thread, probed every two seconds by every replica.
  • A startup probe with too short a grace period, killing a process that needed forty seconds to warm up, forever, so the deploy never completes (Rolling Deployments).
  • Checking a dependency that is not actually required, so an outage in an optional analytics service takes the whole fleet out of rotation.
What can race
  • A request can arrive between readiness flipping false and the load balancer noticing. Draining must continue accepting and completing in-flight work for a period after readiness goes false (Graceful Shutdown).
  • Startup races with traffic: an instance registered before its pool is established receives requests it cannot serve. The startup probe exists to close this window.
  • Concurrent probes can trigger concurrent expensive checks unless the result is cached or the check is deduplicated (Request Coalescing).
Security
  • Health endpoints are typically unauthenticated so probes can reach them. That makes anything they return public — never include versions, dependency hostnames, connection strings or per-check error messages (Not Leaking Your Internals).
  • A detailed health endpoint is a reconnaissance gift: it enumerates your dependencies and tells an attacker when one of them is down, which is when your defences are weakest.
  • Exclude health endpoints from request logging and from your latency metrics, or the constant probe traffic distorts every number you have.
  • An unauthenticated health endpoint that performs a real dependency query is a small amplification primitive: cheap for a caller, work for your database. Cache it (Rate Limiting).
Misreads
  • "Health means everything is working." Health means *this instance* should receive traffic. Whether the system is working is a different question, answered by metrics and SLOs (SLOs: A Target, a Window, and a Reason).
  • "A thorough health check is a better health check." For liveness, thoroughness is the bug. The more it checks, the more ways it has to trigger a restart that fixes nothing.
  • "Readiness and liveness are the same thing with different names." They have opposite consequences: one removes traffic reversibly, the other destroys the process.
  • "If the database is down, we should return unhealthy." Only if nothing in your service works without it. Removing every instance guarantees total unavailability where partial was available.
  • "The orchestrator handles this." The orchestrator executes the probe you configured against the endpoint you wrote. Both are yours (Running a Backend on Kubernetes).
  • "200 means healthy." It means the check passed. A check that returns a constant passes forever, including while the process is deadlocked.

Operating it

How you see it in production
  • Graph readiness state per instance over time. Flapping is obvious in that view and nearly invisible in aggregate metrics.
  • Count restarts per instance. A rising restart count with no deploy is the signature of a liveness misconfiguration, and it is the first thing to check when recovery is taking longer than it should.
  • Track time from process start to ready. A creeping value predicts the day a startup probe threshold becomes too tight.
  • Alert when the number of ready instances falls below what capacity requires — that is the number that matters, not whether any single instance is healthy (Headroom: The Capacity You Deliberately Do Not Use).
  • Log every readiness transition with the reason. "Why was this instance out of rotation at 14:30" is a routine incident question with no other source.
What changes at 10x and 100x
  • At 10x instances, probe traffic itself becomes load. Three endpoints, probed every few seconds, across hundreds of replicas, is a request rate that must not touch a database.
  • At 100x, correlated readiness failures are the real danger: any check that depends on a shared resource makes the whole fleet fail together, which is the opposite of what redundancy is for (Failure Propagation).
  • With autoscaling, readiness gates when new capacity actually helps. A slow ready transition means scaling responds later than the metric that triggered it (Autoscaling a Backend).
  • Startup checks matter more as instance counts grow, because a rolling deploy across hundreds of instances is bounded by how quickly each one becomes ready (Rolling Deployment and the Compatibility It Demands).
What this costs
  • A shallow liveness check will not catch some genuinely wedged states. That is the correct trade: a missed wedge affects one instance, while a false positive can restart the entire fleet.
  • Readiness that ignores dependencies keeps instances in rotation that will return errors. Serving a fast, correct 503 from a live instance is usually better than having no instances at all — but not always, and it depends on whether any of your endpoints work without that dependency.
  • Cached dependency status means readiness reacts a few seconds late. That lag is almost always worth avoiding self-inflicted load on a struggling dependency.
  • Three endpoints is more configuration and more to get wrong. The alternative is one endpoint whose semantics are wrong for at least two of its three consumers.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

  • GENERALThe three questions exist wherever something supervises a process — an orchestrator, a load balancer, a process manager or a deployment platform. Only the names and the mechanism change.
  • CLOUD-SPECIFICKubernetes names them startupProbe, readinessProbe and livenessProbe with explicit failureThreshold and periodSeconds. A cloud load balancer typically has only one health check, which behaves like readiness — so on that platform, liveness has no consumer and putting dependency checks in the one available check is far more dangerous. ECS, App Runner and similar managed products each expose a different subset; check which of the three semantics you actually get before designing the endpoint.
  • RUNTIME-SPECIFICWhat "wedged" means differs, so the legitimate liveness content differs: a blocked event loop on Node means the process cannot serve anything and a loop-lag threshold is a genuine liveness signal; on a threaded runtime one blocked thread is survivable and thread-pool exhaustion is the equivalent condition.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

Domains that do not exist yet
  • Testing & Reliability Engineering — verifying probe behaviour under a simulated dependency outage is a test almost nobody writes, and the restart loop is exactly what it would catch.