The question this answers
What is actually inside a container image, and why does its internal structure change how you build and ship?
The API team rebuilds on every commit. A build that changes one Python file should not reinstall three hundred megabytes of dependencies, should produce the same result on a colleague's machine, and must not silently pick up a newer system library than the one that was tested.
A content-addressed artifact whose composition is inspectable layer by layer, so a build is cacheable, a change is attributable, and two images can be compared byte for byte rather than by hopeful reasoning.
Base image → runtime → dependencies → application → metadata
Every image is assembled in the same order, from the most stable thing to the most volatile. A base image supplies a minimal userland: libc, CA certificates, timezone data, sometimes a shell and package manager. On top of it goes the language runtime — an interpreter, a JVM, or nothing at all for a static binary. Then dependencies: the packages your manifest pins. Then your application code. Finally a configuration metadata document that is not a filesystem layer at all: entrypoint, command, default environment variables, working directory, user, exposed ports, labels.
The order is not stylistic. Each step is a separate layer identified by the hash of its contents, and a build reuses a cached layer only when that layer and every layer beneath it are unchanged. Put COPY . . before the dependency install and you have told the builder that a one-character change to a comment invalidates the dependency install. Put the manifest copy and install first, and the same change reuses everything and rebuilds one small layer.
The metadata deserves attention because it is where a running container gets its defaults. An image with no USER runs as root. An image whose entrypoint is a shell wrapper will not forward SIGTERM to your process, and your graceful shutdown never runs — see Graceful Shutdown: The 502 Spike Nobody Investigates.
/etc, optionally a shell and package manager.libpq, libjpeg, ffmpeg.USER means root; a shell entrypoint swallows signals; a wrong workdir turns into a confusing file-not-found on start.Layers are additive, shared — and permanent
Layers stack through a union filesystem. Reading a path walks down the stack until a layer supplies it; a later layer can *shadow* an earlier one, and a delete is recorded as a whiteout marker rather than as removal. This is the single most misunderstood property of images: deleting a file in a later layer does not remove it from the image. The bytes are still there, still shipped, still extractable, and still counted in the transfer.
That is why a Dockerfile that copies a private key, uses it, then runs rm in a later instruction has published the key to everyone who can pull the image — the topic of Configuration Belongs Outside the Image. It is also why "cleanup" steps rarely shrink anything: the only way to not ship bytes is to never put them in a layer, which is what multi-stage builds exist for (Why Image Size Is an Infrastructure Problem).
The upside of the same design is sharing. Layers are content-addressed, so twenty services built from one base store and pull that base once per host. An image *reference* is either a mutable tag or an immutable digest over the manifest — and the difference between those two decides whether "the same version" means anything (The Container Registry).
LAYER SIZE CREATED BY sha256:9a1c… 74.8MB FROM python:3.11-slim <- base userland sha256:4f02… 61.3MB RUN apt-get install -y libpq-dev <- unpinned: drifts between builds sha256:c7d9… 318.4MB RUN pip install -r requirements <- dominant layer; cache this sha256:2b55… 0.4MB COPY deploy_key /tmp/deploy_key <- present forever sha256:81ae… 0.0MB RUN rm /tmp/deploy_key <- whiteout only; bytes still ship sha256:d340… 1.9MB COPY ./src /app <- the layer that actually changes TOTAL 456.8MB across 6 layers — 1.9MB of it is your code. Config: Entrypoint ["/bin/sh","-c","python app.py"] <- shell wrapper, will not forward SIGTERM Config: User "" <- empty means root
Two Dockerfiles that build the same application
The difference below is not style. The left one rebuilds its dependency layer on every commit, ships a compiler toolchain and a credential it thought it deleted, and runs as root with an entrypoint that cannot be signalled. The right one changes exactly one small layer per commit, and the artifact contains only what the process needs at runtime.
Read it as a rule: order instructions from least to most frequently changed, pin what you install, and never let a secret touch a layer.
FROM python:3.11 WORKDIR /app COPY . . # any change busts everything below RUN apt-get update && apt-get install -y build-essential libpq-dev RUN pip install -r requirements.txt # reinstalled on every build COPY deploy_key /tmp/deploy_key RUN pip install -r private-reqs.txt && rm /tmp/deploy_key CMD python app.py # shell form: PID 1 is sh, SIGTERM is swallowed
FROM python:3.11-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential=12.9 libpq-dev=15.4-0+deb12u1 && rm -rf /var/lib/apt/lists/*
COPY requirements.txt . # only the manifest — cache hits until it changes
RUN pip install --prefix=/install -r requirements.txt
FROM python:3.11-slim # runtime stage: no compiler, no build cache
RUN useradd --system --uid 10001 app
COPY --from=build /install /usr/local
COPY ./src /app # last and smallest: the only layer that changes
WORKDIR /app
USER 10001
ENTRYPOINT ["python", "app.py"] # exec form: your process is PID 1Cache order decides build time; stage separation decides what ships; USER and the exec-form entrypoint decide whether the container is safe and shuts down cleanly. The credential is absent rather than deleted, which is the only version that works.
Key points
- An image is ordered content-addressed layers plus a metadata document — entrypoint, user, env defaults — that is not itself a layer.
- Build caching is layer-by-layer and prefix-sensitive: order instructions from least to most volatile or you rebuild everything, every time.
- Deleting a file in a later layer does not remove it from the image. A secret that touched a layer has shipped.
- Layers are shared across images built from the same base, which is why a common base is a real disk and bandwidth saving.
- A tag is a mutable pointer; a digest is the artifact. Only the digest makes "the same image" a verifiable claim.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Each build instruction that changes the filesystem produces a new layer: a tar of the changes, identified by the hash of its contents.
- • The manifest lists those layer digests plus the config document; the manifest's own digest is the image's immutable identity.
- • The builder reuses a cached layer when the instruction and every preceding layer are unchanged — the cache is a prefix match, not a per-line match.
- • At run time the runtime mounts the layers as a union filesystem, lowest first, and adds one writable layer on top for the container.
- • A deletion is stored as a whiteout entry in the upper layer; the shadowed bytes remain in the lower layer and remain in the transfer.
- • You own the base image lifecycle: pick one, pin it by digest, and rebuild deliberately on a schedule so patches land on purpose rather than by surprise.
- • You own instruction order. It is the cheapest build-time optimization available and it costs nothing but attention.
- • You own the metadata: a non-root
USER, an exec-form entrypoint, a correct workdir, and labels recording the source commit. - • You own build reproducibility — pinned system packages and a committed lockfile, or "the same Dockerfile" quietly means "a different image".
- • A rebuild with no code change produces a different image because an unpinned base or
apt-getpulled newer packages; the regression is invisible in the diff. - • Every build takes eleven minutes because
COPY . .precedes the dependency install and the cache never hits. - • A secret is committed to a layer, "removed" in the next instruction, and extracted from the published image months later.
- • The container ignores
SIGTERMbecause a shell is PID 1, so every rollout ends in a forced kill and dropped in-flight requests. - • The image runs as root because no
USERwas set, turning a mounted host path into a host-level compromise.
- • Build time scales with how much of the layer stack is invalidated, not with repository size — good ordering keeps it near-constant as the project grows.
- • Registry storage scales with distinct layers, not with tags: a thousand builds sharing one base store that base once.
- • Pull time on a cold host scales with the layers the host is missing, which is why a shared base across services pays off repeatedly.
- • Layer count itself has a ceiling in most runtimes; hundreds of tiny layers slow mounting and buy nothing.
- • Every layer under your code is third-party software you are shipping and running. The base image is a supply-chain dependency — see The Infrastructure Supply Chain.
- • Layer contents are readable by anyone who can pull the image. Treat an image as publishable even when the registry is private.
- • Scan images for known vulnerabilities at build and again on a schedule, because the vulnerability set changes while the image does not.
- • A smaller base removes attack surface directly: no shell and no package manager means no shell and no package manager for an attacker either.
- • Build minutes are driven by cache hit rate, which is driven by instruction order — the cheapest cost lever in the whole pipeline.
- • Registry storage is driven by distinct layers and by how long a retention policy keeps old tags.
- • Transfer is driven by uncached layers times pull count, which is where image composition becomes a network bill — see Why Image Size Is an Infrastructure Problem.
- • Build duration split by cache hit and miss, which tells you exactly which instruction is in the wrong place.
- • Image size and layer count per build, tracked over time — silent growth is how a 200MB image becomes a 2GB one.
- • Vulnerability scan results per layer, so you know whether a finding is yours or the base's.
- • The signal that lies: a green build. It says the instructions succeeded, not that the resulting artifact is the one you tested last week.
- • A single static binary in a
scratchor distroless image — no base userland, nothing to patch, nothing to exploit. If your language allows it, this is strictly simpler. - • A buildpack or platform builder that generates the image from source. You lose fine control and gain a maintained, patched, sensibly-layered base for free.
- • For a small internal tool, a plain tarball on a VM is still a legitimate artifact. Layer mechanics only pay for themselves when builds are frequent.
- • Layer caching buys fast builds and costs discipline: the fastest ordering is rarely the most readable one.
- • A shared base buys pull and storage savings and costs coupling — one bad base rebuild affects every service at once.
- • A minimal base buys a smaller attack surface and costs debuggability: no shell in the image means no shell when you need to look inside a production container.
- • Pinning by digest buys reproducibility and costs a recurring update chore that someone has to actually do.
Image layers: what size actually costs you
each layer is content-addressed and cached independently a change invalidates its layer AND every layer after it so the ordering rule is: least-volatile first, most-volatile last FROM base → install deps (from the lockfile alone) → COPY source → CMD
What people believe, and what is true
Removing a file in a later instruction makes the image smaller.
It adds a whiteout entry. The bytes stay in the lower layer and still transfer on every pull.
The same Dockerfile always produces the same image.
Only if the base and every install are pinned. An unpinned FROM and an unpinned apt-get make the build a function of the day it ran.
Fewer layers is always better.
Collapsing everything into one instruction destroys the cache and makes every build reinstall everything. Layer *boundaries* are the tool; use them where change frequency changes.