Draining: Stopping Without Dropping
Stop new connections, let active work finish, then exit. The whole difficulty is ordering — the instance must leave the routing layer before it stops serving, and those two events are not naturally sequenced.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does an instance stop serving without dropping the requests that were already in flight?
Instances are removed constantly — every deploy, every scale-in, every node replacement — and each removal is a chance to drop work that was already accepted.
Send the process a termination signal. It shuts down, the platform notices, and traffic goes to the other instances.
The order is wrong and it is wrong by default. The termination signal and the routing removal are usually issued at the same time by different components, so for a window the instance is shutting down and still receiving new requests.
- The order is wrong and it is wrong by default. The termination signal and the routing removal are usually issued at the same time by different components, so for a window the instance is shutting down and still receiving new requests.
- A process that exits promptly on the signal abandons every request it was in the middle of. Those requests were accepted and will not be answered — they are errors attributable to your deploy, not to any fault (Graceful Shutdown: The 502 Spike Nobody Investigates in the backend view).
- Routing removal is not instant anywhere. It propagates through a controller, a dataplane, a load balancer and possibly a client-side cache, each on its own schedule (Service Discovery in Operation).
- Long-lived connections do not end just because you stopped accepting new ones. A streaming connection, a websocket or a persistent HTTP connection will happily continue past any grace period you configured (WebSockets in the networking view).
- Background work is invisible to all of this: a request may have returned while a job it started is still running, and the shutdown path usually does not know about it (Background Jobs and Workers in the backend view).
- When the grace period expires, the process is killed outright. Everything still running is lost, and the grace period was chosen by someone who did not measure the longest request.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A correct drain is a sequence: stop being routed → stop accepting new work → finish what is in flight → release resources → exit. Reversing any two of those steps drops requests.
- The critical ordering problem is between the first two. On most platforms the removal from routing and the termination signal are issued concurrently, so a process that begins refusing connections immediately will refuse traffic that is still being sent to it.
- The standard fix is a delay at the start of shutdown: on receiving the signal, keep serving normally for long enough that routing removal has propagated, and only then begin the drain. In Kubernetes this is a
preStophook; on a VM fleet it is the deregistration delay before the instance is stopped. - Failing readiness is the explicit way to leave routing while still serving — it removes the instance from the endpoint list without stopping anything (Probes: Readiness, Liveness and Startup).
- The grace period is a hard ceiling. When it expires the process is killed with a signal it cannot catch, exactly like an OOM kill, and in-flight work is lost (OOMKilled: Over the Memory Limit).
- Long-lived protocols need an application-level goodbye — a protocol-level message telling the client to reconnect elsewhere — because there is no transport-level way to say "finish up" to an open stream.
The sequence, and the one that everyone gets wrong
Five steps, and their order is the entire lesson. The first step looks like doing nothing, which is why it is the one that gets removed when someone is making deploys faster.
- 1Leave routing
Fail readiness or deregister, so the routing layer stops selecting this instance.
fails by Skipped entirely — the process starts shutting down while it is still in the endpoint list.
evidence The instance is absent from the service's endpoints or target group.
- 2Wait for propagation
Keep serving normally while removal reaches every dataplane and cache.
fails by Omitted, because it looks like an unnecessary sleep. This is where the dropped requests come from (Service Discovery in Operation).
evidence Incoming request rate to this instance falls to zero before it stops accepting.
- 3Stop accepting new work
Refuse new connections; stop pulling from queues and streams.
fails by HTTP stops and the queue consumer keeps pulling messages it will not finish (Operating Queues and Scheduled Work).
evidence No new work is accepted at any entry point.
- 4Finish in-flight work
Let active requests and jobs complete.
fails by Exiting immediately on the signal, abandoning everything in progress.
evidence Active request count reaches zero before exit.
- 5Release and exit
Close connections and pools, flush buffered telemetry, exit cleanly.
fails by Killed at the grace-period ceiling with work still running.
evidence Clean exit before the grace period, with no forced termination.
Steps one and two happen while the instance is completely healthy and still serving. That is what makes them feel wrong, and it is exactly why they work.
The race, in order
This is the default behaviour on a platform where routing removal and the termination signal are issued concurrently. Nothing here is misconfigured in the usual sense — the timeline is what you get if you do nothing.
- T+0changeRolling update decides to replace this pod
- T+0changeThe endpoint controller is told to remove it, and the kubelet is told to terminate it — concurrently
- T+0actionThe process receives the termination signal and immediately stops accepting new connections
- T+0signalRouting removal has not yet reached every node's dataplane; requests are still being sent here
- T+1signalThose requests are refused. At the edge they appear as connection errors on a small fraction of traffic
- T+1actionThe process finishes its in-flight work and exits cleanly, reporting a graceful shutdown
- T+1recoveryRouting removal finishes propagating. No further requests are sent here
- AftersignalEvery deploy produces the same small error burst. It correlates with deploys, so it is attributed to the changes rather than to the shutdown
- FixrecoveryA pre-stop delay is added: on the signal, keep serving normally until removal has propagated, then drain. The error burst disappears
The process did everything right and still dropped requests. The bug is in the ordering between two components, which is why it cannot be fixed inside the application alone.
What escapes the drain
A drain that handles HTTP correctly can still lose work, because several kinds of work are not requests. These are the ones that are routinely missed.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Queue consumer | Messages redelivered or lost at every deploy | The consumer kept pulling after HTTP stopped, and was killed holding unacknowledged messages | Stop consuming first, finish what is held, then exit (Operating Queues and Scheduled Work) |
| Background job started by a request | Work silently unfinished; the request returned successfully | The drain tracks requests, not the jobs they spawned | Make such work durable and restartable rather than in-process (Background Jobs and Workers in the backend view) |
| Websocket or stream | Clients disconnected abruptly during rollouts | Long-lived connections outlive any grace period | Send a protocol-level close and let clients reconnect with jitter |
| Long upload or export | Killed at the grace-period ceiling | The grace period was set from typical requests, not the longest ones | Set it from the real maximum, or move the work off the request path |
| Buffered telemetry | Missing logs and metrics for the final moments before every shutdown | Buffers never flushed | Flush during release; these are the logs you will want in a postmortem (Reconstructing What Actually Happened) |
| Client holding a direct connection | Errors after the instance left every routing layer | A pooled connection bypasses discovery entirely | Bound connection lifetimes on the client side (Service Discovery in Operation) |
| Spot or preemptible reclamation | Much less warning than a deploy provides | The platform's notice period is shorter than your grace period | Know the notice period; do not run work longer than it on reclaimable capacity |
How to do it properly
Most important first.
- Order the steps deliberately: delay first so routing catches up, then stop accepting, then drain, then exit. The delay is doing real work even though it looks like sleeping.
- Set the grace period from the longest request the service actually serves, including uploads and long-running operations — and know what that is rather than guessing (Long-Running Operations: 202 and the Job Resource in the API view).
- Handle the termination signal explicitly in the application. A process that ignores it is killed at the end of the grace period every single time (Signals: Asynchronous Notifications From the Kernel in the operating systems view).
- Stop accepting new work at every entry point, not just HTTP: queue consumers, schedulers and stream subscriptions all need to stop taking new work and finish what they hold (Operating Queues and Scheduled Work).
- For long-lived connections, send a protocol-level close and let clients reconnect, with jitter so they do not all reconnect to the same replacement at once (Thundering Herd in the concurrency view).
- Measure the drain by errors during deploys. If a rolling update produces a small burst of errors every time, draining is misconfigured and the burst is the evidence (A Successful Deploy Is Not Evidence of a Healthy System).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The rolling update contains it: only traffic on the instances being replaced is exposed, so a broken drain costs a small fraction of requests per deploy rather than an outage. That containment is also why it goes unfixed for years — the errors are real, attributable and small enough to be averaged away on a dashboard. It stops being one-percent during a mass event: a node replacement, a zone drain or a scale-in that removes many instances at once.
What can go wrong
- No delay before shutdown, so the instance refuses connections that routing is still sending to it — the default behaviour on most platforms, and the most common cause of deploy-time errors.
- Grace period shorter than the longest request, so slow requests are killed at the ceiling on every deploy.
- The application ignoring the termination signal entirely, making every shutdown a hard kill.
- Draining HTTP while queue consumers keep pulling new messages, so the process is killed with unacknowledged work in hand (Dead Letter Queues Are an Operation).
- Long-lived connections outliving the grace period, so a rollout silently disconnects streaming clients.
- A drain so long that a rolling update takes far longer than expected and a rollback is correspondingly slow when you need it to be fast.
- Draining that assumes the load balancer is the only route, while a client-side cache or an established connection pool keeps sending directly (Connection Pooling in the networking view).
- "The platform handles graceful shutdown." It sends a signal and waits. What the process does with the signal, and whether routing has caught up, are yours.
- "A short grace period is safer." It is faster. It converts in-flight requests into errors at the ceiling.
- "Draining is a deploy concern." Every scale-in, eviction, node replacement and spot reclamation drains an instance, and those happen without anyone deploying (Autoscaling).
- "We do not see errors during deploys, so draining is fine." Check whether you would see them: a brief burst on a fraction of requests is easy to average away (Percentiles: Which One, and How Many Users Is That? in the observability view).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Error rate at the edge flat across a rolling update — the definitive test, because a drain problem shows up as a burst on every deploy.
- Requests still being served by an instance after it left the endpoint list, and completing successfully.
- No containers terminated at the grace-period ceiling: a kill at the ceiling means work was still running when time ran out.
- Long-lived connection counts falling smoothly during a rollout rather than dropping to zero at a boundary.
- Draining configuration is part of the workload definition, so changing it is a rollout — and the first rollout after the change is also the test of it.
- Dropped requests cannot be recovered. Anything that must survive a drain has to be safe to retry, which pushes the real protection into idempotency rather than into shutdown timing (Idempotency in Backends in the backend view).
- If a long drain is making rollbacks too slow during an incident, reducing the grace period is available and it trades correctness for speed — a deliberate choice worth naming when you make it.
- Automate the whole sequence in the platform rather than in each service. The delay, the signal handling contract and the grace period are exactly what a service template should provide (Service Templates).
- Automate the check: errors during deploys, attributable to draining, should be a standing signal rather than something noticed by a user (Deploys on the Same Timeline as the Symptom).
- Keep the grace period a per-service decision, because it depends on the longest legitimate request and only the owning team knows that.
- A long grace period drops fewer requests and makes every rollout and rollback slower, which matters most in the incident where you need the rollback to be fast.
- The pre-shutdown delay costs time on every single pod replacement across the fleet, in exchange for removing a class of error nobody was attributing correctly.
- Perfect draining for long-lived connections requires protocol support and client cooperation, which is real work in both the server and every client (WebSocket Message Contracts in the API view).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICEndpoint removal and the termination signal are issued concurrently by different components, which is precisely why a
preStopdelay is standard practice rather than a workaround. A cloud load balancer with a deregistration delay sequences this for you — the instance is removed from the target group first and stopped after — so the VM equivalent is a setting rather than a hook. A PaaS usually does the sequencing for you and does not tell you what its grace period is. - GENERALThe five-step sequence and the ordering requirement apply to any system where routing membership and process lifetime are managed separately, including queue consumers and stream processors that have no load balancer at all.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.