Graceful Shutdown
Signal, stop accepting work, drain what is in flight, release resources, exit — inside a hard timeout you do not control.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What must a container do between receiving a termination signal and exiting, so that a rollout does not drop user requests?
Containers stop constantly — every rollout, every scale-down, every node drain, every eviction, every spot reclamation. If stopping is abrupt, then a small amount of user-visible failure is built into every one of those events, permanently.
The process exits when it is told to. The load balancer notices the instance is gone and stops sending traffic. Any request that was in flight will be retried by the client.
The load balancer notices afterwards. Between the process exiting and the routing layer removing it, requests are still being sent to an address with nothing listening, and they fail (Operating a Load Balancer).
- The load balancer notices afterwards. Between the process exiting and the routing layer removing it, requests are still being sent to an address with nothing listening, and they fail (Operating a Load Balancer).
- On an orchestrated platform the termination signal and the endpoint removal are concurrent, not ordered. The signal often arrives before routing has converged, so requests continue to arrive for a period after the application has been told to stop.
- "The client will retry" assumes an idempotent operation and a client that retries. A non-idempotent write that fails halfway is not recoverable by retry, and a browser request is not retried at all (Version Coexistence: N and N+1, in Both Directions).
- Keep-alive connections are already established. Removing the instance from a load balancer's pool does not close them, so an existing connection keeps delivering requests until it is closed by one side (Keep-Alive and Connection Reuse in networking terms).
- Background work is invisible to this whole discussion: a queue consumer holding a message, a job halfway through, a batch mid-write. Nothing about HTTP-shaped draining addresses them, and they lose work silently (Operating Queues and Scheduled Work).
- Without a bounded drain the process may never exit, so the platform kills it and every stop costs the full grace period — which makes rollback slow at exactly the moment speed matters.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The sequence has five steps and they must be in this order: receive the signal, stop accepting new work, drain what is in flight, release resources, exit — all within a hard timeout after which the platform sends SIGKILL (PID 1 and Signals).
- Step two is not "stop serving". It is stop taking *new* work while continuing to serve what is already in progress, and — critically — usually continuing to accept new requests for a short period, because the routing layer has not converged yet. Failing readiness immediately while still serving is the shape that works (Probes: Readiness, Liveness and Startup).
- The convergence gap is the part that surprises people. Endpoint removal propagates through the control plane, the load balancer and any client-side load balancing at their own rates. A brief sleep before beginning the drain — or a pre-stop hook — gives that propagation time to complete while the process is still able to serve.
- Draining means different things per work type. In-flight HTTP requests are finished and their connections closed. Established keep-alive connections are told not to reuse the connection and are closed after the current response. Queue consumers stop fetching and finish or explicitly release the message they hold. Long-lived streams are closed with a signal the client understands as "reconnect".
- Releasing resources means the things whose absence causes a *different* failure later: flushing buffered writes and telemetry, closing database connections so the pool on the other side is freed, releasing locks and leases so another instance can take them (The Connection Budget).
- The hard timeout is the constraint that makes this an engineering problem rather than a checklist. Every step must fit inside the platform's grace period, so the drain needs its own shorter bound and a decision about what to abandon when it expires.
The five steps, and what each one is for
The order is the content. Reversing steps two and three — draining before you stop accepting — never terminates. Doing step four before step three closes the connections that in-flight work is using.
Every step also has a cost in time, and the sum of those costs must fit inside a timeout set by the platform, not by you.
- 1Receive the signal
The application's handler runs and enters shutdown state.
fails by The signal never arrives, because a shell is PID 1 or no handler is installed (PID 1 and Signals).
evidence A log line, written by the application, naming the signal.
- 2Stop accepting new work
Fail readiness immediately; keep serving while routing converges; then stop accepting new connections.
fails by Refusing traffic instantly, before the routing layer has stopped sending it — which turns a graceful shutdown into a burst of connection errors (Probes: Readiness, Liveness and Startup).
evidence Readiness reports unready while requests are still being answered successfully.
- 3Drain in-flight work
Finish active requests, close keep-alive connections after their current response, release or complete held queue messages.
fails by An unbounded wait on a connection that never closes, so the drain never ends (Draining: Stopping Without Dropping).
evidence In-flight count reaches zero, logged, within the drain bound.
- 4Release resources
Flush buffered writes and telemetry, close connection pools, release locks and leases.
fails by Done too early, cutting off work that is still running; or skipped, leaving leases held until they expire (The Connection Budget).
evidence Downstream connection counts drop; no lease is held by a dead instance.
- 5Exit
Terminate deliberately, with a code that says what happened.
fails by Never exiting, so the grace period expires and SIGKILL arrives.
evidence Exit code 0 or 143, not 137.
The drain in step three needs its own timeout, shorter than the platform grace period, after which the process abandons what remains and exits anyway — logging what it abandoned. Being killed teaches you nothing; exiting deliberately tells you exactly what did not finish.
The race: the signal arrives before the traffic stops
This is the mechanism behind almost every "small error rate during deploys" investigation, and it is genuinely counter-intuitive: the platform tells the container to stop and keeps sending it traffic, because the two actions are independent and propagate at different rates.
The timeline below uses a configured pre-stop delay and a configured grace period. Those are settings, not measurements — the point is the ordering, and that the drain must not start before routing has converged.
- T+0changePlatform decides to stop this instance. Endpoint removal begins propagating; SIGTERM is sent — the two are concurrent, not ordered.
- T+0actionApplication handler runs, marks itself unready, and deliberately keeps serving.
- T+0 to pre-stop delaysignalRequests continue to arrive from load balancers that have not yet converged. They are served normally, because the process is still listening.
- after pre-stop delayactionRouting has converged; new requests stop arriving. The listener is closed.
- drain windowactionIn-flight requests complete. Keep-alive connections are closed after their current response. Queue messages in hand are finished or released.
- in-flight reaches zeroactionBuffers and telemetry flushed; connection pools closed; leases released.
- before grace period expiryrecoveryProcess exits deliberately, code 0. No request failed.
- grace period expirysignalThe alternative path: if the process is still running, SIGKILL. In-flight work is lost with no record of what it was.
The first three rows are the whole insight. An application that stops listening at T+0 fails every request that arrives during the convergence window, and there is nothing wrong with the network.
Where drains go wrong
The handler below is deliberately unremarkable — the ordering and the timeout are the content, not the API. Note that the drain has its own bound and exits regardless, which is the difference between a shutdown that is graceful and one that merely intends to be.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Listener closed at T+0 | Connection-refused errors on every rollout | Drain started before routing converged | Fail readiness first, keep serving through a pre-stop delay |
| Unbounded drain | Every stop takes exactly the grace period; exit 137 | Waiting on a connection that never closes | Bound the drain and exit deliberately, logging what was abandoned |
| Pool closed first | Burst of database errors in the last seconds of every instance | Resource release ordered before in-flight completion | Release after the drain, not at the start of the handler |
| Consumer holding a message | Occasional duplicate processing after deploys | Killed while a message was leased | Release or finish the message during the drain; make handlers idempotent (Operating Queues and Scheduled Work) |
| Websocket clients | Stops never complete within the grace period | Long-lived connections have no natural end | Send a close with a reconnect instruction, then bound the wait |
| Grace period raised to compensate | Node drains and evictions become slow; rollbacks take longer | A drain problem hidden by a longer timeout | Fix the drain; the grace period should follow the measured drain, not lead it |
| Spot reclamation | Work lost despite a correct drain | The warning was shorter than the drain needs | Make the workload interruptible as well as drainable (Reducing Blast Radius) |
1let ready = true2const drainBudgetMs = graceSeconds * 1000 - preStopMs - 2000 // leave room to exit3 4process.on('SIGTERM', async () => {5 log.info('sigterm: entering shutdown')6 ready = false // 1. readiness fails immediately7 await sleep(preStopMs) // 2. keep serving while routing converges8 server.close() // 3. stop accepting new connections9 10 const drained = await Promise.race([11 waitForInflight(), // finish active requests12 timeout(drainBudgetMs), // ...but not forever13 ])14 if (!drained) log.warn('drain timeout', { inflight: inflightCount() })15 16 await consumer.stop() // 4. release held queue messages17 await telemetry.flush()18 await db.close() // after in-flight work, never before19 log.info('shutdown complete')20 process.exit(0) // 5. exit deliberately21})Three things are doing the work here and none of them are the framework. The readiness flag flips before the sleep, so propagation and serving overlap. The drain has a budget derived from the grace period rather than a fixed number. And the pool is closed after in-flight work rather than at the top, which is the single most common ordering mistake.
How to do it properly
Most important first.
- Handle the termination signal explicitly and start the sequence. In the exec form, so the signal actually arrives (PID 1 and Signals).
- Fail readiness first, keep serving second. The instance should report itself unready immediately and continue answering requests until the routing layer has caught up.
- Give routing convergence a deliberate delay — a pre-stop hook or a sleep at the top of the handler — sized to your platform's propagation, not guessed.
- Stop the server from accepting new connections, then wait for in-flight requests with a bound that is comfortably shorter than the grace period.
- Handle each work type separately: HTTP in-flight, keep-alive connections, background jobs, queue messages, streams. A drain that only covers HTTP will lose the others silently.
- Release in the right order — stop work, then flush, then close pools and release leases, then exit. Closing the database pool before in-flight requests finish converts a graceful shutdown into a burst of errors.
- Set the grace period from the measured drain, and make sure the drain's own timeout is shorter. Exit deliberately when it expires, logging what was abandoned, rather than being killed.
- Make the whole thing observable: log entry into shutdown, the count of in-flight work, and the exit reason (Using Observability, Not Building It).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rolling deployment bounds the affected instances at any moment, so the loss is a thin continuous stream rather than an outage — which is exactly why it survives for years unnoticed. Nothing contains a node-level event: a drain or an eviction stops every container on that node at once (Rolling: Two Versions, One Database).
What can go wrong
- Requests arriving after the process stopped listening, because the drain began before routing converged. This is the single most common cause of deploy-correlated errors.
- A drain that waits on a long-lived connection that never closes — a websocket, a server-sent event stream, a long poll — so every stop takes the full grace period.
- The database pool closed while requests are still in flight, turning a clean shutdown into a burst of connection errors in the last seconds.
- A queue consumer that exits while holding a message, so the message is redelivered and processed twice — which is only safe if the handler is idempotent (Dead Letter Queues Are an Operation).
- Readiness failing but the process exiting at the same moment, which removes the benefit entirely: the point of failing readiness early is to keep serving while it propagates.
- The mitigation failing in the most damaging way: a graceful shutdown handler with no timeout. Every stop now takes the full grace period, so rollouts are slow and rollbacks are slow, which lengthens every incident (Rollback: Only Useful If It Is Actually Safe).
- A grace period raised to accommodate a slow drain, which also raises the time a node drain or an eviction takes, and therefore the time a failing node keeps failing.
- "We handle SIGTERM, so we are done." Handling it is step one of five. A handler that immediately exits is a faster abrupt shutdown, not a graceful one.
- "Failing readiness stops the traffic." It starts the process of stopping the traffic. Propagation takes time, and the whole point of the drain is to keep serving during it.
- "Retries make this unnecessary." Retries help idempotent requests with retrying clients. They do not help non-idempotent writes, browser navigations, or a job that was half-done.
- "A longer grace period is safer." It is safer for draining and worse for every situation where you want the container gone quickly — evictions, node drains, rollbacks during an incident.
- "This only matters at high traffic." It matters wherever a dropped request matters. At low traffic the absolute number is small and the probability that any given deploy loses someone's request is unchanged.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Error rate during a rollout is indistinguishable from error rate between rollouts, measured at the client-facing edge rather than at the service (A Successful Deploy Is Not Evidence of a Healthy System).
- Containers exit with code 0 or 143 during normal rollouts, and 137 is rare enough to be investigated when it happens.
- Stop duration tracks the actual in-flight work rather than sitting at exactly the grace period every time.
- Shutdown logs show the sequence: unready, drain started, in-flight count reaching zero, resources released, exit.
- A deliberate node drain in a lower environment completes without any request failing, rehearsed rather than assumed.
- A broken shutdown handler is reverted by deploying the previous digest, and this is one of the cases where you find out quickly, because the symptom appears on the very next rollout.
- The deeper problem is circular: if shutdown is broken, rollback itself is slow and lossy, because rolling back stops containers too. Getting shutdown right is a prerequisite for the recovery mechanism you rely on for everything else.
- Work abandoned at SIGKILL is not recoverable by rolling back. Whatever was in flight is gone, and only idempotency and retry on the caller's side recover it (Roll Forward: When Going Back Is the Harder Option).
- Automate a shutdown test in CI: start the image under load, send the termination signal, assert zero failed requests and a bounded exit.
- Automate the pre-stop delay and the grace period as platform configuration held with the service, so they are reviewed like code rather than set once by hand (Infrastructure as Code).
- Do not automate an increase in the grace period as a response to slow shutdowns. That hides the drain problem and slows every eviction and node drain in exchange.
- A longer grace period makes drains reliable and makes every stop slower in the worst case — including rollbacks, node drains and spot reclamations, where slow is genuinely costly.
- A pre-stop delay adds fixed time to every container stop in exchange for eliminating the convergence race. It is usually the best trade available and it is not free.
- Draining long-lived connections properly requires the client to handle a reconnect instruction, which is application protocol work outside the container (WebSockets in networking terms).
- Perfect draining is impossible against a hard kill. Some events — a node failure, a spot reclamation with a short warning — give less time than a drain needs, so the workload must also be safe to interrupt (Reducing Blast Radius).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe five-step sequence and the hard-timeout constraint hold on every platform that stops containers, which is all of them. What each step means depends on the work: an HTTP service drains requests, a consumer drains messages, a stream server drains connections.
- PLATFORM-SPECIFICThree values differ and all three matter: the stop signal, the grace period, and whether a pre-stop hook exists. Kubernetes sends SIGTERM, honours a per-pod termination grace period, offers a pre-stop hook, and removes the endpoint concurrently with the signal. A managed container service may fix the grace period and offer no hook. A serverless container platform may drain for you and give the instance no opportunity to run a handler at all. Establish your platform's three values before designing the drain around them.
- SIMPLIFIEDThe timeline in this lesson shows one instance and one routing layer. Real topologies have several — a client-side load balancer, a mesh sidecar, an ingress and a cloud load balancer — each converging at its own rate, so the effective convergence window is the slowest of them rather than the one you configured.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — a drain that has never been exercised under load is a design, not a behaviour.