The question this answers
What must a process do between being told to stop and actually stopping, so that no request in progress is lost?
A deploy, a scale-down or a node replacement must not fail a request that had already been accepted. A user who pressed submit must get an answer, whether or not the instance handling it was chosen for replacement two hundred milliseconds later.
An ordered termination: the instance stops being eligible for new traffic, completes work it already accepted, releases resources cleanly, and exits — with a bounded grace period after which it is killed anyway.
The sequence, and the two places it goes wrong
When an orchestrator or autoscaler decides an instance should go, it sends a termination signal — SIGTERM on Unix — and starts a timer. When the timer expires it sends SIGKILL, which cannot be caught. Everything useful happens in between, and the default behaviour of most runtimes for an uncaught SIGTERM is to exit immediately. Every connection is severed mid-response. From the client's side those requests become connection resets or, through a proxy, 502s.
The correct sequence has four steps. Stop accepting: fail the readiness probe and stop the listener accepting new connections, so the load balancer removes this instance from rotation. Finish: let in-flight requests complete, and for a worker, stop taking new jobs and finish the current one. Close: flush buffers, commit or return in-progress work, close database connections, deregister from service discovery. Exit: with a zero status, before the grace period runs out.
The second failure is subtler and more common than the first: the race between failing the readiness probe and the load balancer noticing. Readiness is polled — every few seconds — and the load balancer needs another moment to propagate the change. If the process stops accepting connections the instant it receives SIGTERM, there is a window of several seconds during which the balancer still routes traffic to a socket that is closed. The fix is counterintuitive and important: on SIGTERM, mark yourself unready but keep serving for a few seconds, then begin draining. You deliberately spend part of the grace period doing nothing except letting the routing layer catch up.
- 1SIGTERM received
The platform has decided this instance goes. A grace-period timer starts; SIGKILL follows when it expires.
The default handler exits immediately, dropping every in-flight request. This is the signature bug.
- 2Fail readiness, keep serving5–15 seconds — ILLUSTRATIVE
The readiness endpoint starts returning failure, while the listener stays open. Wait a few seconds for the load balancer to remove this target.
Skipping this. Closing the listener at once means the balancer keeps routing to a closed socket for several seconds — the same 502s, from the other direction.
- 3Drain in-flight work
Stop accepting new connections; let accepted requests finish. Workers stop claiming jobs and finish the current one.
A long-running request exceeding the grace period. Cap request duration, or the timer will cut it off anyway.
- 4Close resources
Flush logs and metrics, commit or nack in-flight messages, close database connections, deregister from service discovery.
Unflushed telemetry, so the last minutes before the shutdown — exactly the interesting part — are missing.
- 5Exit 0
The process exits cleanly, well inside the grace period.
Exceeding the grace period: SIGKILL, and the drain is truncated with in-flight requests still open.
The signature: a small spike of 502s at every deploy
It looks like this. Error rate is a flat 0.05%. During each deploy it rises to 0.4% for ninety seconds and then returns. Nobody pages, because the alert threshold is above it and it resolves on its own. It is mentioned once in a retrospective, described as "just the deploy", and nobody investigates because there is no single failing request to look at — only a hundred of them, spread across every batch boundary of the rollout.
The arithmetic makes it worth caring about. A service at 500 requests per second replacing twenty instances, each dropping roughly two hundred milliseconds of in-flight work, drops on the order of two thousand requests per deploy. At ten deploys a week that is twenty thousand failed requests a week from one preventable bug, and each one is a real user seeing a real error.
It is also diagnostically distinctive: the errors correlate exactly with deploy windows, appear at the proxy rather than in application logs, and show as connection resets rather than as application errors. If your 5xx graph has a comb pattern whose teeth line up with your deploy history, this is what you are looking at — and the same pattern appears on scale-down and on node replacement, which is why it also shows up at 04:00 when the autoscaler shrinks the fleet.
BEFORE — process exits on SIGTERM immediately 12:00:00 deploy starts, batch 1 of 5 12:00:00 instance-04 SIGTERM -> exit(0) in 3ms 12:00:00 lb: 41 in-flight requests to instance-04 reset 12:00:02 lb: still routing to instance-04 (readiness poll interval) -> connection refused 12:00:04 edge 502 count: 137 ...repeats at every batch boundary... 12:01:30 deploy complete. total 502s: 1,904 AFTER — handler with an unready pause, then drain 12:00:00 deploy starts, batch 1 of 5 12:00:00 instance-04 SIGTERM -> readiness now failing, listener STILL OPEN 12:00:00 lb: 39 in-flight requests continue; new requests still accepted for now 12:00:08 lb: target removed from rotation (poll + propagation) 12:00:08 instance-04 stops accepting; 6 requests still in flight 12:00:09 instance-04 in-flight complete; connections closed; metrics flushed 12:00:09 instance-04 exit(0) (grace period 30s — finished with 21s to spare) 12:01:40 deploy complete. total 502s: 0
Implementing it, including the parts that are not the HTTP server
The HTTP side is the easy half and every mainstream framework supports it. The parts that get forgotten are the other resources: a message consumer that must stop claiming work and finish or return the message it holds, a background scheduler that must not start a new run, a metrics client whose buffer must be flushed, and a database connection pool that should be closed after the last query rather than before it.
The message consumer case deserves attention because getting it wrong causes duplicates rather than errors. A worker killed mid-job leaves its message unacknowledged; the broker redelivers it; the job runs twice. If the job is not idempotent, that is a double charge or a duplicate email, and it happens on every deploy. Graceful shutdown for a worker means: stop claiming, finish or explicitly return the current message, then exit — and the work should be idempotent anyway, because SIGKILL exists and no handler survives it.
Finally, size the grace period against real request durations. A grace period shorter than your slowest request guarantees truncation; one much longer than needed makes every deploy slow and every node drain slower. Measure the p99 of request duration, add the readiness propagation delay, and set the period comfortably above the sum. And keep the handler simple: a shutdown path that can itself hang is worse than none, because it turns a fast failure into a slow one.
1const GRACE_MS = 30_000 // must be under the platform termination grace period2const LB_PROPAGATION_MS = 8_000 // readiness poll interval + propagation, measured not guessed3 4let ready = true5app.get('/readyz', (_req, res) => res.status(ready ? 200 : 503).end())6 7process.on('SIGTERM', async () => {8 // 1. Stop being eligible for NEW traffic, but keep serving.9 ready = false10 log.info('sigterm: readiness off, still serving')11 12 // 2. Wait for the load balancer to notice. Skipping this is the second-13 // most common cause of deploy 502s, after not handling SIGTERM at all.14 await sleep(LB_PROPAGATION_MS)15 16 // 3. Stop accepting; let accepted requests finish.17 await new Promise<void>((resolve) => server.close(() => resolve()))18 19 // 4. The parts that are not HTTP.20 await consumer.stop() // stop claiming; finish or nack the message in hand21 await scheduler.stop() // do not begin another run22 await metrics.flush() // or the last minutes are missing from the graphs23 await db.end() // after the last query, not before24 25 process.exit(0)26})27 28// A shutdown path that can hang is worse than none: bound it.29process.on('SIGTERM', () => setTimeout(() => process.exit(1), GRACE_MS).unref())Key points
- Stop accepting → finish in-flight → close resources → exit, inside a bounded grace period ending in SIGKILL.
- The default behaviour for an uncaught SIGTERM in most runtimes is immediate exit, which drops every in-flight request.
- Fail readiness first and keep serving for several seconds, so the load balancer stops routing before the listener closes.
- The signature is a small, regular 502 spike correlating exactly with deploys, scale-downs and node replacements.
- Workers must stop claiming and finish or return the message in hand, or every deploy redelivers jobs and duplicates side effects.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The platform sends SIGTERM and starts a termination grace-period timer; SIGKILL follows when it expires.
- • The application marks itself unready; the readiness probe fails on its next poll.
- • The load balancer or service removes the instance from its target set after the poll and propagation delay.
- • The application stops accepting new connections and lets accepted requests complete.
- • Resources are released in dependency order — consumers, schedulers, telemetry, then connection pools.
- • The process exits with status zero, well before the grace period ends.
- • You own the grace period, sized against measured p99 request duration plus readiness propagation.
- • You own the readiness endpoint's semantics and its poll interval, since the two together set the propagation delay.
- • You own the non-HTTP resources: consumers, schedulers, telemetry buffers and pools.
- • You own capping request duration; a request that can run longer than the grace period will be truncated regardless of the handler.
- • You own testing it, which means sending SIGTERM under load in a pre-production environment and counting the failures.
- • No handler at all: immediate exit, every in-flight request reset, a 502 spike at every batch boundary.
- • Closing the listener immediately on SIGTERM: the load balancer keeps routing for several seconds to a closed socket, producing the same errors from the other side.
- • A grace period shorter than the slowest request, so drains are truncated by SIGKILL.
- • A shutdown handler that hangs — waiting on a connection that will never close — turning a fast termination into a grace-period-long one.
- • A worker killed mid-job, so the broker redelivers and a non-idempotent side effect happens twice on every deploy.
- • Telemetry not flushed, so the minutes before shutdown — precisely the ones you want during an investigation — are absent.
- • Long-lived connections (WebSocket, streaming) held open through the grace period and then severed, with no reconnect guidance sent to the client.
- • Total dropped requests scale with instance count times request rate times drop window, which is why the problem grows quietly as the fleet grows.
- • Deploy duration scales with grace period times the number of batches, so an over-long grace period makes every rollout and every node drain slower.
- • Spot and preemptible instances make this acute: terminations are frequent and the notice period is short and fixed, so the handler must complete inside it.
- • Autoscaler scale-down hits the same path continuously, which is why the 502 comb often appears at times when nobody deployed.
- • An abandoned in-flight request may have completed a side effect without returning a result, which is exactly the ambiguity idempotency keys exist to resolve.
- • Unflushed audit logs at shutdown mean the record of the last actions before termination is missing — a real gap in an investigation.
- • A truncated drain can leave a transaction open until the database times it out, holding locks and blocking other work.
- • Credentials and tokens held in memory are released with the process; there is no additional cleanup required, but a handler that writes diagnostic state to disk on shutdown may leave them behind.
- • Essentially free to implement — a handler and a correctly sized grace period.
- • The indirect cost is deploy duration: grace period times batches, which on a large fleet with a long period is real wall-clock time.
- • The cost being avoided is failed requests times deploy frequency, which for an active service is a substantial number that never appears on any invoice.
- • 5xx count at the edge during deploy windows specifically, compared with the surrounding baseline. This is the diagnostic.
- • Connection reset counts at the load balancer, which distinguish a dropped connection from an application error.
- • Time from SIGTERM to exit per instance, and how often SIGKILL was reached — a rising SIGKILL rate means the grace period is now too short.
- • Message redelivery rate correlated with deploys, which is the worker-side version of the same bug.
- • The signal that lies: application error logs. The dropped requests never produced one — the process was gone. The evidence is at the proxy, not in the app.
- • Connection draining at the load balancer alone, which handles requests already dispatched but does nothing about work the application accepted internally. It is a partial mitigation, not a substitute.
- • A pre-stop hook that sleeps before the signal reaches the process, which buys the propagation delay without touching application code. Useful for services you cannot modify; it does not drain in-flight work.
- • Idempotent operations plus client retries, which make dropped requests recoverable rather than prevented. This is worth having anyway, and it is not a reason to skip the handler.
- • For a genuinely stateless service with millisecond-length requests and retrying clients, the impact may be small enough to accept — measure it before assuming so, because most teams overestimate how short their p99 is.
- • Buys zero dropped requests during termination; costs a longer deploy and a handler to maintain in every service.
- • A longer grace period buys safety for slow requests and costs wall-clock time on every rollout and every node drain.
- • The deliberate unready pause buys correctness against the routing layer and costs seconds spent doing nothing, which reads as waste until you see the 502 graph.
What people believe, and what is true
The load balancer drains connections, so the application does not need to.
Draining stops new dispatches. Work the application already accepted — an in-flight request, a claimed message, a running job — is still lost when the process exits.
Fail the readiness probe and close the listener immediately.
Readiness is polled and propagation takes seconds. Closing at once means the balancer routes to a closed socket, producing the same 502s. Stay open for a few seconds first.
It is just a few errors during the deploy.
It is request rate times drop window times instances times deploys per week, every one of them a real user. It is small per deploy and large per quarter, which is why it survives.