What Image Size Actually Costs
Size is paid on cold pulls and nowhere else — and an image with no shell is a real operational cost that nobody puts on the other side of the ledger.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does a large image actually cost, and when is making it smaller the wrong optimisation?
Image size is highly visible, easy to measure and easy to compare, so it attracts optimisation effort out of proportion to what it costs — while the thing it trades against, operability during an incident, is invisible until an incident.
Smaller is better. Move to the smallest possible base — distroless, or scratch with a static binary — strip everything the application does not strictly need, and treat the size number as a quality metric.
The cost is paid on a cold pull: a node that does not already have the layers must fetch the ones it is missing. On a node that already has them, image size costs nothing at start-up.
- The cost is paid on a cold pull: a node that does not already have the layers must fetch the ones it is missing. On a node that already has them, image size costs nothing at start-up.
- That means the number to reason about is not the image size but the size of the layers that are *not already on the node* — which is dominated by how often your top layers change and how widely your base is shared, not by the total (Layers and the Build Cache).
- Image size is not memory. A large image does not make a running process consume more RAM; the read-only layers are on disk and paged in as needed (Memory Pressure, Swap and the OOM Killer in OS terms).
- Aggressive minimisation trades away debuggability. An image with no shell, no package manager and no diagnostic tools is exactly the image you cannot inspect at 3am, and the platform features that substitute for it may not exist where you are running (Debugging a Container in Production).
- Some minimal bases change runtime behaviour rather than merely removing files: a different libc implementation can differ in DNS resolution behaviour, thread stack defaults and locale handling. That is a correctness risk accepted in exchange for size, and it is rarely stated as such (DNS Failure Modes: What Each One Looks Like is the OS-side view of the first one).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Pull cost is proportional to the bytes the node does not already have, and it is incurred at cold start: first deploy to a node, node replacement, autoscaling, eviction and rescheduling, and any pull-policy setting that forces a re-fetch.
- Because layers are shared and content-addressed, two services on a common base share those blobs on every node that runs both. Standardising the base across an organisation reduces real pull volume more reliably than shrinking any individual image (Artifact Registries).
- Registry storage and transfer costs scale with distinct blobs, not with image count — again favouring shared bases over individually minimal ones (Egress: Moving Data Costs Money, Not Just Storing It in cloud terms).
- Size correlates with scan surface: more packages means more findings, most of which are in components your application never invokes. That is a triage cost rather than a risk measurement, and treating it as risk is the error the domain specifically warns about (Scanning, and Why a Finding Is Not a Risk).
- What you remove for size is also what you have during an incident. A shell, a package manager,
ps, a DNS lookup tool and a way to make an HTTP request are the difference between diagnosing inside the failing container and inferring from outside it.
Where size costs, and where it does not
Half of this table is the part people act on and the other half is the part they assume. The assumptions are what produce effort spent on a number that is not connected to anything.
| Situation | Does size cost? | What actually drives it |
|---|---|---|
| Deploy to nodes that already ran a previous version | Barely | Only the changed top layers transfer |
| First deploy to a new node pool | Yes | Every layer transfers, base included |
| Autoscaling on new nodes | Yes, on the critical path | Cold pull is part of time-to-capacity (Autoscaling) |
| Node replacement or eviction | Yes | Same as a new node, at an unpredictable moment |
| Scale-to-zero / serverless cold start | Yes, as user-visible latency | Pull time is inside the request path (Scale to Zero) |
| Steady-state memory use | No | Process allocation, unrelated to image size |
| Steady-state CPU | No | Unrelated |
| Registry storage bill | Yes, for distinct blobs | Shared base layers are stored once (Artifact Registries) |
| Vulnerability findings count | Yes | More packages, more findings — not necessarily more risk |
| Incident diagnosis | Inversely | What you removed is what you do not have (Debugging a Container in Production) |
Choosing a runtime base
There is no correct answer here, only a trade made with the cold-pull frequency and the debugging story in view at the same time. The failure is making it with only one of them in view.
What should the runtime stage's base image be?
when Nodes are long-lived, cold pulls are rare, and in-place diagnosis matters more than bytes.
cost Largest pull on cold nodes, largest scan surface, a shell available to anything that gets code execution.
when The common default: same libc and tooling ecosystem, most of the size gone.
cost Some diagnostic tools missing; usually installable during an incident if a package manager remains.
when Size matters and you have verified the runtime behaves identically under load.
cost A different libc. DNS resolution behaviour, thread stack defaults and some binary compatibility differ — verify rather than assume (How Networks Fail in Production).
when Cold pulls are frequent, the security posture benefits, and your platform supports attaching a debug container.
cost No shell and no package manager. Diagnosis depends entirely on a platform feature; if it is unavailable, you are debugging by redeploy.
when A single self-contained binary with no runtime dependencies at all.
cost You must supply CA certificates and timezone data yourself, and the failure when you forget is a TLS or date error in production. No diagnosis is possible inside the container.
The counter-warning, stated plainly
Every optimisation in this lesson has a mirror image, and the mirror is only visible during an incident. An image with no shell is not a smaller image with the same properties — it is an image you cannot look inside.
This is not an argument against minimal images. It is an argument for making the trade with both sides written down: what the size buys, and what the missing tooling costs when something is wrong and nobody knows why.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Incident on a distroless image | Cannot exec; no shell exists | Runtime base has no shell and the platform offers no debug container here | Reproduce locally with the identical digest and config; add the debug-container capability as the action item (Debugging a Container in Production) |
| Static binary on scratch | Every outbound HTTPS call fails on certificate verification | No CA certificate bundle in the image | Copy the CA bundle in the final stage (Multi-Stage Builds) |
| Static binary on scratch | Timestamps wrong or timezone lookups fail | No timezone database in the image | Copy the tz data, or keep everything in UTC (Production Time Is UTC) |
| Base swapped to musl for size | Intermittent resolution failures under concurrency | Different resolver behaviour than the previous libc | Verify under real concurrency before rollout; treat a base change as a change (DNS Failure Modes: What Each One Looks Like in OS terms) |
| Size gate added to CI | Debug tooling removed from images to pass the gate | A trade-off encoded as a pass/fail threshold | Report size, do not gate on it (Guardrails, Not Gates) |
| Separate debug image maintained | The debug image does not reproduce the problem | It drifted from the production image | Debug the production digest itself; a divergent debug image is worse than none (Parity That Is Worth Paying For) |
How to do it properly
Most important first.
- Establish where the cost actually lands for you before optimising: are your nodes long-lived and warm, or replaced constantly? Autoscaling and spot-instance fleets pull far more often than a static fleet.
- Use multi-stage builds to leave the build toolchain behind. This is the change with the best ratio of size removed to operability lost (Multi-Stage Builds).
- Standardise on a small number of base images across services so nodes already hold the common layers.
- Keep the volatile part of the image in the top layers so a new version transfers only the application, not the runtime.
- Decide debuggability explicitly, and write the decision down. A distroless runtime image is a fine choice if you have a working way to attach a debug container or reproduce locally with an identical image (Debugging a Container in Production).
- Where cold-start latency genuinely matters — scale-to-zero, serverless, burst autoscaling — size becomes a latency problem rather than a cost problem, and the trade shifts (Scale to Zero).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A pull that is too slow contains itself to nodes starting up, so a warm fleet is unaffected and a scaling event is not. What is not contained is a base image change that alters runtime behaviour — that ships to everything, and the rollout strategy is the only limit (Canary: One Percent, Then Five, Then Watch).
What can go wrong
- A minimal image in an incident: no shell, no way to exec, and the platform's ephemeral debug container feature unavailable or unfamiliar, so diagnosis happens by redeploying with logging added.
- A base image swap made for size that changes DNS or TLS behaviour under load, producing an intermittent failure that looks like a network problem.
- Statically linked binaries that lack CA certificates or timezone data, failing on the first outbound TLS call or the first date formatting in production (Multi-Stage Builds).
- Size optimisation that reorders layers and destroys cache locality, making every build slower to make every pull marginally faster.
- Chasing the total size number while the top layer — the one that changes every deploy — stays large, so pull cost per deploy is unchanged.
- The mitigation failing: a "debug variant" image maintained alongside the minimal one, which drifts until it is a different piece of software and reproduces nothing.
- "Our image is large, so our service uses a lot of memory." Unrelated. Image size is disk and network; process memory is what the process allocates (OOMKilled: Over the Memory Limit).
- "We cut the image in half, so deploys are twice as fast." Only for the layers the nodes did not already have. If the base was already present, halving the base changes nothing about your deploys.
- "Fewer packages means fewer vulnerabilities." It means fewer findings. Whether it means less risk depends on whether the removed packages were reachable, which the scanner does not know (Scanning, and Why a Finding Is Not a Risk).
- "Alpine is just a smaller Debian." It uses a different libc, and the differences show up in DNS resolution behaviour, thread defaults and some binary compatibility. It is a good choice made deliberately and a bad one made for size alone.
- "We do not need a shell; we have logs." Logs answer the questions you thought to ask before the incident. The shell answers the ones you did not.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Cold-start behaviour on a genuinely empty node is measured on your platform, not assumed from the size number.
- The distribution of layer sizes is known, and the layer that changes every deploy is small relative to the rest.
- Someone has successfully diagnosed a problem inside a production-shaped container using whatever mechanism the minimal image leaves available — rehearsed, not theorised.
- After a base image change, the service is exercised under real concurrency before it carries traffic, because behaviour differences show up under load rather than at start-up.
- Reverting an image-size change is a redeploy of the previous digest, which is cheap (Rollback: Only Useful If It Is Actually Safe).
- What is not cheap is reverting a base image change that has already been depended upon — application code adapted to the new base, or a fix that only exists on top of it. Base changes should ship on their own, not bundled with features (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Automate size reporting as information: layer breakdown per build, and the delta from the previous build.
- Automate the parts that are pure win — build-stage separation, ignore files, and not shipping build caches.
- Do not automate a size threshold that fails the build. It converts a trade-off into a gate, and the pressure it creates is relieved by removing debugging tools, which is the wrong side of the trade.
- Smaller images pull faster on cold nodes and remove the tools you would want during an incident. This is the central trade and it should be made per service, based on how often the service cold-starts and how it is debugged.
- A shared base image reduces pull volume across the fleet and couples every service to one upgrade decision.
- Distroless and scratch improve the security posture by removing a shell an attacker could use, and remove the same shell an operator could use. Whether that is a good trade depends entirely on whether your platform gives you an alternative (Production Debugging).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICHow much size costs is decided by node churn, which is a platform property. A serverless container platform or a scale-to-zero configuration pulls on nearly every request burst, so size is directly user-visible latency. A static VM fleet with long-lived nodes pulls on deploy and rarely otherwise, so the same image size costs almost nothing. A spot-instance or heavily autoscaled fleet sits in between and moves with your traffic pattern.
- GENERALThe mechanism — cost is proportional to bytes not already present on the node, incurred at cold pull — holds everywhere. Only the frequency of cold pulls differs.
- TOOL-SPECIFICThe debuggability escape hatch depends on the runtime. Kubernetes offers ephemeral debug containers that attach to a running pod's namespaces, which makes a shell-less image workable; a bare container runtime, a managed container service or a serverless platform may offer nothing equivalent, in which case a shell-less image means no in-place diagnosis at all. Check what your platform actually supports before removing the shell.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.