PID 1 and Signals
The entrypoint becomes PID 1, PID 1 does not get default signal handling, and a shell wrapper in between is why your container ignores SIGTERM.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does my container ignore the stop signal and take the full grace period to die every single time?
Every deploy stops containers. If the termination signal does not reach the application, every deploy kills it abruptly, and the resulting dropped requests look like a load balancer problem.
The container runtime asks the container to stop and it stops. If it takes a while, the application must be slow to shut down — probably something in the code holding it open.
The entrypoint process becomes PID 1 inside the container's PID namespace, and the kernel treats PID 1 specially: a signal for which the process has installed no handler is not delivered with its default action. A program that never handles SIGTERM therefore does not die on SIGTERM — it ignores it, without any code that says so.
- The entrypoint process becomes PID 1 inside the container's PID namespace, and the kernel treats PID 1 specially: a signal for which the process has installed no handler is not delivered with its default action. A program that never handles SIGTERM therefore does not die on SIGTERM — it ignores it, without any code that says so.
- The runtime then waits out the grace period and sends SIGKILL, which cannot be ignored. The application receives no notice, drops whatever it was doing, and exits with code 137.
- Wrapping the entrypoint in a shell makes it worse: the shell is PID 1, the application is a child, and a shell running a command in the common non-
execform does not forward signals to it. The signal reaches the shell, the shell ignores it for the PID 1 reason above, and the application never learns anything happened. - A process that spawns children and does not reap them accumulates zombies, because PID 1 is the namespace's reaper and a normal application was not written to do that job.
- The symptom is generic and misattributed. "Some 502s during deploys" gets investigated at the load balancer, in the service mesh, or in the application's request handling, when the cause is that the process is being killed rather than asked to stop.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- When a container starts, the runtime execs the entrypoint inside a new PID namespace. That process is PID 1 there, and it inherits the responsibilities of an init process: it receives orphaned children to reap, and it gets the kernel's PID 1 signal treatment (Creating Processes: fork, exec, wait in OS terms).
- The kernel rule, precisely: for a signal other than SIGKILL and SIGSTOP, the kernel will not apply the default action to a namespace's PID 1 if that process has not installed a handler for it. Signals from within the namespace are blocked outright; signals from an ancestor namespace — which is where the container runtime lives — are also not delivered, with SIGKILL and SIGSTOP the forced exceptions.
- That is exactly the sequence a stop performs: SIGTERM first, then SIGKILL after the grace period. An unhandled SIGTERM is a no-op, so every stop takes the full grace period and ends in a kill (Signals: Asynchronous Notifications From the Kernel in OS terms).
- The Dockerfile
CMD/ENTRYPOINTexec form — a JSON array — execs the binary directly, so the application is PID 1 and any handler it installs works. The shell form — a bare string — runs it under/bin/sh -c, inserting a shell as PID 1. - A wrapper script is fine as long as its last line is
execthe real process:execreplaces the shell rather than forking, so the application takes over PID 1 and the shell disappears. - A minimal init — tini, dumb-init, or the runtime's own init option — solves both problems at once: it installs handlers, forwards signals to the process group, and reaps orphans. It costs a small binary and is the right default whenever the container has a process tree rather than a single process.
Four entrypoints, four outcomes
All four of these start the same application. Only two of them let it hear a termination signal, and the difference is entirely in the form of the instruction.
The second form is the most common accident: it looks like the first, it is a string rather than an array, and that alone inserts a shell between the runtime and the application.
1# 1. exec form: the app is PID 1. Its SIGTERM handler runs.2ENTRYPOINT ["node", "server.js"]3 4# 2. shell form: /bin/sh -c is PID 1, node is its child.5# The shell does not forward SIGTERM, and as PID 1 with no6# handler it ignores it. Grace period elapses, SIGKILL, 137.7ENTRYPOINT node server.js8 9# 3. wrapper without exec: same problem, one file further away.10# entrypoint.sh: ./migrate && node server.js11ENTRYPOINT ["/entrypoint.sh"]12 13# 4. wrapper with exec: the shell is replaced, the app is PID 1.14# entrypoint.sh: ./migrate && exec node server.js15ENTRYPOINT ["/entrypoint.sh"]Forms 3 and 4 are the same Dockerfile line; the difference is one word inside the script. That is what makes this failure so durable — the Dockerfile review passes, and the behaviour is decided by a file nobody opened.
What the stop sequence actually does
Reading the sequence as steps rather than as "the runtime stops it" makes the failure point obvious: exactly one step can silently do nothing, and every later step assumes it worked.
- 1Stop requested
A rollout, scale-down, node drain or eviction decides this container should end.
fails by Nothing yet — but note how many ordinary operations reach this step.
evidence A stop event in the runtime or orchestrator log.
- 2Signal sent to PID 1
The runtime sends the configured stop signal, usually SIGTERM, to the container's PID 1.
fails by PID 1 has no handler, so the kernel does not apply the default action and nothing happens.
evidence The application logs that it received the signal.
- 3Forwarding
If PID 1 is a wrapper or an init, it must pass the signal to the real process.
fails by A shell that does not forward, which is the default for
sh -c.evidence The application, not just the wrapper, logs the signal.
- 4Application drain
Stop accepting work, finish in-flight work, release resources.
fails by No handler, or a drain with no timeout that never finishes (Graceful Shutdown).
evidence In-flight requests complete; new ones are refused or routed elsewhere.
- 5Exit
The process exits, ending the container.
fails by Exit never happens within the grace period.
evidence Exit code 0, or 143 if the process chose to exit on the signal.
- 6Grace period expiry
The runtime sends SIGKILL, which cannot be handled or ignored.
fails by This step running at all is the failure — it means the earlier steps did not work.
evidence Exit code 137, and work that was in flight is simply gone.
Two exit codes distinguish the whole story: 143 means the process saw the signal and acted; 137 means it was killed. A service that always exits 137 on deploy has never once drained.
Signal failures and how they present
None of these present as "signal handling is broken". They present as deploy-correlated errors, slow rollouts, or a process count that grows all week.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Shell-form entrypoint | Every stop takes the full grace period; exit 137 | A shell is PID 1 and neither forwards nor dies | Use exec form, or exec the process in the wrapper |
| No SIGTERM handler in the app | Same symptom, exec form in place | PID 1 with no handler gets no default action | Install a handler that starts the drain (Graceful Shutdown) |
| App spawns unreaped children | Zombie count grows; eventually a process limit is hit | PID 1 is the reaper and the app is not an init | Add a minimal init, or reap in the application |
| Init forwards to the process group | Helper processes die before the main drain finishes | Group-wide signal delivery during shutdown | Forward to the main process only, or make helpers drain-aware |
| Handler with no timeout | Every stop takes the full grace period, exit 137, drain logs present | The drain waits on something that never completes | Bound the drain below the grace period and exit anyway (Graceful Shutdown) |
| Grace period shorter than the drain | Drain starts, is cut off mid-way, exit 137 | Platform timeout below what the workload needs | Raise the grace period, or shorten the drain — and know both numbers |
| Exit 137 assumed to be OOM | Memory limits raised repeatedly with no effect | 137 is SIGKILL from any source | Check the OOM indicator before tuning memory (OOMKilled: Over the Memory Limit) |
How to do it properly
Most important first.
- Use the exec form for
ENTRYPOINTandCMD. This is the single highest-value line in this lesson. - If an entrypoint script is necessary, end it with
exec "$@"orexec /app/server, so the application replaces the shell. - Install a SIGTERM handler in the application and make it start the shutdown sequence rather than exit immediately (Graceful Shutdown).
- Use an init process when the container legitimately runs more than one process, or when the application spawns children it does not reap.
- Verify rather than assume: send the termination signal locally and watch for the application's own shutdown log line. A container that dies instantly with code 143 handled it; one that takes the whole grace period and exits 137 did not.
- Know your platform's grace period and set it deliberately — it is the hard deadline every drain has to fit inside.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rolling deployment limits the damage to the instances being replaced at any moment, so the error rate is small and continuous rather than total. Nothing contains the second-order effect: every stop taking the full grace period lengthens rollback, which lengthens every incident (Rolling: Two Versions, One Database).
What can go wrong
- Every deploy takes the full grace period per instance, which lengthens rollouts and, more importantly, lengthens rollbacks during an incident (Rollback: Only Useful If It Is Actually Safe).
- Requests dropped on every rollout, presenting as a small, consistent error rate correlated with deploys and nothing else (Deploys on the Same Timeline as the Symptom).
- Zombie processes accumulating until the container hits its process limit, in a container that spawns subprocesses per unit of work.
- A signal reaching the shell and not the application, so the application's carefully written shutdown handler never runs and everyone believes it is broken.
- An init process added that forwards signals to the process group and starts killing helper processes the application needed during its own drain.
- The mitigation failing: a SIGTERM handler that starts a drain with no timeout, so the process never exits and the platform kills it anyway — with the added cost that it now takes the full grace period every time (Graceful Shutdown).
- "The application ignores SIGTERM, so it must trap it somewhere." It almost certainly does not. The kernel is declining to apply the default action because the process is PID 1 and has no handler. Nothing in the application says "ignore".
- "Exit code 137 means out of memory." It means the process was SIGKILLed — 128 plus 9. An OOM kill is one cause and a grace-period timeout is another, and they need different fixes. Check the OOM indicator, not the code alone (OOMKilled: Over the Memory Limit).
- "Adding tini fixes shutdown." It fixes signal delivery and reaping. If the application still has no SIGTERM handler, the signal now arrives and nothing happens with it.
- "This only matters for long-running requests." It matters for every rollout, every scale-down, every node drain, every spot reclamation — all of which stop containers, several times a day.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The application logs its own shutdown message when stopped, and the container exits with code 143 rather than 137.
- Stopping a container takes about as long as the drain should take, not exactly the grace period every time.
- Error rate during a rollout is indistinguishable from error rate between rollouts.
- A process listing inside a long-running container shows no accumulating zombies.
- Reverting an entrypoint change is a redeploy of the previous digest, and it is safe (Tags Versus Digests).
- The uncomfortable case is discovering this during an incident: a broken process model means the rollback itself is slow, because every instance takes the full grace period to stop. Fixing the process model is therefore a prerequisite for fast recovery, not an optimisation on top of it.
- Automate the check in CI: start the image, send the termination signal, assert the process exits cleanly within a bound and with a handled exit code. It is a short test and it catches the whole class.
- Automate the exec form via a lint rule on the Dockerfile — shell-form entrypoints are mechanically detectable.
- Do not automate away the application's own handler. Only the application knows what draining means for it; an init that kills the process on SIGTERM is correct behaviour for the init and destroys the drain.
- The exec form loses shell features in the entrypoint — variable expansion, globbing, chained commands. The answer is an entrypoint script that ends in
exec, which is one more file and keeps both properties. - An init process is another binary in the image and another thing to keep current, in exchange for correct reaping and signal forwarding.
- A long grace period gives drains room and makes every stop slow in the worst case; a short one makes stops fast and drains impossible. There is no setting that avoids the trade (Draining: Stopping Without Dropping).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALPID 1 signal semantics are a Linux kernel property of PID namespaces, so they apply to every container runtime on Linux identically. The Dockerfile exec-versus-shell form distinction is an OCI image config property and is equally universal.
- PLATFORM-SPECIFICWhat differs is the stop sequence around it: which signal is sent first (configurable per image and per platform), how long the grace period is and where it is configured, and whether the platform offers a pre-stop hook that runs before the signal. Kubernetes sends SIGTERM, honours a per-pod termination grace period and supports a pre-stop hook; a bare runtime has a default timeout and no hook; a managed container service may not let you change either. Find your platform's three values before tuning anything.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.