Debugging a Container in Production
What survives a container's death, what does not, and the order to ask questions in when there is no shell and the evidence is being deleted on every restart.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The container is crash-looping and the image has no shell. What do you actually look at, and in what order?
Container failures destroy their own evidence: the writable layer is discarded, the process is gone, and a restart policy is busy overwriting the state you needed while you decide what to do.
Exec into the container and look around — check the config, check the logs, check whether the file it wants is there.
A crash-looping container is not running when you get there. There is nothing to exec into, and the one that existed a moment ago has been removed along with everything in it.
- A crash-looping container is not running when you get there. There is nothing to exec into, and the one that existed a moment ago has been removed along with everything in it.
- A minimal image has no shell to exec into even when it is running, which is a trade made deliberately elsewhere and paid for here (What Image Size Actually Costs).
- Restarts overwrite. On many platforms only the current and previous container's output is retained, so a fast crash loop can discard the first, most informative failure before anyone looks.
- Logs written to a file inside the container die with it, which means the most detailed record of the failure is the one guaranteed to be unavailable (Image Versus Container).
- A large share of container failures are not in the application at all. They are at the boundary — a missing environment variable, an unmountable volume, a wrong user, an unreachable dependency — and none of those produce application logs, because the application never started.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Evidence divides sharply into what lives outside the container and what dies with it. Outside: the runtime's events, the exit code and any kill reason, the image config, the applied spec, and anything the process wrote to stdout that the platform collected. Inside and lost: the writable layer, the process state, anything written to a file.
- The exit code is the cheapest discriminator available. Zero means it decided to stop. A small non-zero code is the application's own choice. 143 is 128+15, SIGTERM handled. 137 is 128+9, SIGKILL — which has more than one cause, and the platform's OOM indicator is what separates a memory kill from a grace-period kill (OOMKilled: Over the Memory Limit).
- Runtime events cover the steps before the application exists — pull, create, mount, start — which is exactly the region application logs cannot describe (The Container Lifecycle).
- The image config is data you can read without running anything: entrypoint, user, working directory, environment defaults, exposed ports. Comparing it with the applied spec resolves a surprising number of "it works locally" cases.
- For a shell-less image, the escape hatch is a debug container that shares the failing container's namespaces — its own filesystem with tools, the target's process, network and, where supported, filesystem view. This is a platform capability rather than a container one, and its availability should be established before you need it.
- The most reliable fallback is reproduction: run the exact digest locally with the same configuration and see whether it fails the same way. If it does, you have an artifact problem; if it does not, you have an environment problem, and that split alone halves the search space.
What survives, and what is already gone
The first column is what you can still read. The second is what disappeared when the container was removed, which on a crash loop is happening repeatedly while you work.
The design implication is on the last two rows: anything you want during an incident must leave the container while it is healthy, because nothing leaves it afterwards.
| Evidence | Survives the container? | Where to get it |
|---|---|---|
| Exit code and kill reason | Yes | Runtime or orchestrator state for the terminated container |
| Lifecycle events (pull, mount, start) | Yes | Runtime or orchestrator event stream (The Container Lifecycle) |
| Stdout and stderr already collected | Yes | The platform's log pipeline |
| Previous container's output | Sometimes, briefly | A previous-logs option; may be overwritten by the next restart |
| Image config: entrypoint, user, env defaults | Yes | Inspect the image by digest; no need to run anything |
| The applied spec and its config references | Yes | Read it back from the platform, not from the repository |
| Log files written inside the container | No | Gone with the writable layer — do not write them (Image Versus Container) |
| Process state, open descriptors, memory | No | Only while running, and only with a debug path |
| Anything written to the container filesystem | No | Discarded on removal |
The order to ask questions in
Ordered by cost, cheapest first. Each step either identifies the failure or narrows it, and the first four require nothing to be running — which matters, because on a crash loop nothing is.
- 1Exit code and kill reason
Separates a deliberate exit, an application error, an OOM kill and a grace-period kill.
fails by Reading 137 as memory without checking the OOM indicator (OOMKilled: Over the Memory Limit).
evidence A code and, for 137, an explicit OOM flag or its absence.
- 2Runtime events
Covers pull, create, mount and start — everything before the application exists.
fails by Skipping straight to application logs for a failure that happened before the application ran.
evidence An event naming the step that failed.
- 3Previous container output
Shows what the last incarnation said before dying.
fails by Being overwritten by the next restart while you are still reading.
evidence The application's own last words, captured somewhere durable.
- 4Image config versus applied spec
Compares what the image expects with what the environment supplied.
fails by Reading the repository instead of the applied spec, which may differ (Configuration Drift).
evidence Entrypoint, user, env and mounts as actually applied.
- 5Reproduce with the same digest
Splits artifact problems from environment problems.
fails by Running a locally-built image with the same tag rather than the failing digest (Tags Versus Digests).
evidence Fails identically (artifact) or does not (environment) — either answer halves the search.
- 6Attach a debug container
Puts tools alongside the failing process, sharing its namespaces.
fails by No such capability on this platform, or wrong namespace sharing so you inspect yourself.
evidence A process listing, a network check or a filesystem view of the target.
Notice that a shell is needed only at the last step, and that the first five have already localised most failures. That is the honest counterweight to "we cannot debug a distroless image" — it is a real cost, and it is smaller than it feels.
Getting a shell into an image without one
When the first five steps have not resolved it and you need to look inside, these are the options. They are not equivalent, and the right one depends on what your platform supports and how much you are willing to change the failing system.
The image has no shell and you need to look inside. What now?
when The platform supports attaching a container that shares the target's namespaces.
cost A privileged operation that needs access controls, and it changes nothing about the target — which is also its advantage (Break-Glass Access).
when You control the workload spec and can redeploy with a debug sidecar.
cost A redeploy, which restarts the failing container and destroys the state you were inspecting.
when The failure is likely in the artifact, or no debug path exists.
cost Requires reproducing the configuration faithfully; an environment-only failure will not reproduce, though that is itself informative.
when The container will not stay up long enough to inspect and the workload is already down.
cost The workload stays down deliberately, and it is a manual change to a running system that must be recorded (Manual Production Changes).
when You have node access and the question is about the host — disk, mounts, kernel logs.
cost Broad privilege, usually the most tightly controlled access there is (Production Access).
when Everything above is exhausted and the failure is reproducible.
cost A new artifact and a build cycle, and it changes the system under investigation. Reasonable as a last resort, common as a first one.
How to do it properly
Most important first.
- Ask in this order: what is the exit code and was it OOM, what do the runtime events say, what did the previous container log, what does the image config say, does it reproduce with the same digest and config.
- Preserve the evidence before you change anything. On a crash loop, capture the previous container's output first — the next restart may remove it.
- Log to stdout and stderr, always, so output escapes the container while it is still running (Using Observability, Not Building It).
- Establish the debug-container path for your platform in advance and rehearse it once, in a lower environment, on a shell-less image (Runbooks).
- Slow the crash loop deliberately when you need to look: increasing a backoff, or temporarily replacing the entrypoint with a sleep, buys inspection time at the cost of the workload being down — which it already is.
- When the image has no shell and no debug path exists, reproduce locally with the same digest rather than rebuilding a "debug variant", which will differ from what is failing (Parity That Is Worth Paying For).
- Resist the reflex to add logging and redeploy as the first move. It is a new artifact, it takes a build cycle, and it changes the thing you are investigating.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Investigation itself is usually read-only and contained to the workload in question. What is not contained is the impulse to act — an exec-and-fix diverges the fleet, and an automated remediation applied to an unexplained failure can widen it (Manual Production Changes).
What can go wrong
- Evidence destroyed by a restart before it was read, which is the default outcome unless someone captures it deliberately.
- Debugging the wrong artifact: a locally-built image with the same tag rather than the failing digest (Tags Versus Digests).
- A fix deployed based on a guess, which changes the symptom without explaining it and removes the failing state you were investigating.
- Exec-ing in and changing something, which fixes one container, diverges the fleet, and reverts silently later (Image Versus Container).
- Attributing 137 to memory and raising limits repeatedly, when the kill was a grace-period timeout (PID 1 and Signals).
- The mitigation failing: a debug container attached with the wrong namespace sharing, so it inspects its own environment and reports that everything is fine.
- "There are no logs, so the application did not log anything." More likely the application never started, or it logged to a file inside a container that no longer exists.
- "Exit 137 means out of memory." It means SIGKILL. Memory is one cause; an unhandled termination signal reaching the grace period is another, and they need opposite fixes.
- "It works locally, so the image is fine." The image is one of three inputs. Configuration, mounted secrets and the environment's dependencies are the other two, and they are where the boundary failures live.
- "We should add more logging." Sometimes. But it is a new artifact and a build cycle, and it changes the system you are investigating. Exhaust what already exists first.
- "Restarting it fixed the problem." It reset the state. The problem is what put the state there (Production Anti-Patterns).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can state the exit code, the kill reason and the lifecycle step for a failing workload within a minute of being paged.
- The previous container's output is retrievable after a restart, and was captured before the loop overwrote it.
- A shell-less image has been successfully inspected in a lower environment using the platform's debug mechanism — done once, deliberately, before it was needed.
- The digest being debugged is confirmed identical to the digest that is failing in production.
- If the failing workload is a new version, rolling back is usually the right first action and the investigation continues on the artifact afterwards. Stopping user impact does not require understanding the cause (Stop the Harm Before You Understand It).
- Capture evidence before rolling back — exit code, events, previous logs — because the rollback removes the failing containers and with them the state that explains the failure.
- If the failure predates the current version, rollback will not help and will cost time. The exit code and the events usually say which case you are in before you have to guess.
- Automate evidence collection on crash: capture exit code, kill reason, events and the previous container's output into somewhere durable, automatically, on every restart.
- Automate the version and digest surface so the identity questions are a lookup (From Developer to Users).
- Do not automate remediation of an unexplained crash loop. An automatic redeploy or an automatic limit increase destroys evidence and can convert a contained failure into a wider one (The Automation Trap).
- Preserving evidence — a longer backoff, a paused container, a retained previous state — extends the outage of that instance in exchange for being able to explain it.
- A minimal image improves the security posture and the pull cost and removes in-place diagnosis. Whether that trade is correct depends entirely on whether a debug-container path exists on your platform (What Image Size Actually Costs).
- Debug containers with broad namespace access are powerful and are a privileged production capability that needs the same controls as any other (Break-Glass Access).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICThe debug path is the single biggest difference between platforms. Kubernetes offers ephemeral debug containers that join a running pod's namespaces, which makes a shell-less image workable. A bare container runtime lets you run a second container sharing the target's namespaces manually. A managed container service or a serverless container platform may offer neither, in which case a shell-less image means no in-place diagnosis at all and reproduction is the only path. Establish which you have before minimising an image.
- TOOL-SPECIFICHow much history the runtime keeps differs: some keep the previous container's output and some do not, some retain exited containers until pruned and some remove them immediately, and the OOM indicator is reported in different places. Whether the first crash in a loop is still readable is decided by these settings, not by how quickly you responded.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.