Containers & Images

Containers in Operation

What containerizing a workload actually changes for the people who run it: one immutable artifact carries the application, its runtime and its dependencies, so the bytes that passed CI are the bytes production starts.

The question this answers

Infrastructure question

What does packaging a workload as a container actually change for the team that has to run it in production?

Application requirement

The checkout service needs Python 3.11, a pinned OpenSSL and three system libraries. It works on the developer's laptop, fails on the CI runner with a missing libpq, and fails differently on the staging VM whose base image drifted two patch releases ahead of production.

What it provides

A single immutable artifact that carries the application, its language runtime and every dependency below it, so "works on my machine" and "works in production" become the same claim about the same bytes.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

The deployable unit becomes the artifact, not the machine

Before containers, a deployment was a *diff applied to a machine*: copy the code, run the package manager, restart the service, hope the machine still resembles the one you tested against. Every machine accumulated its own history — a hotfix applied at 3am, a library upgraded by an unattended-upgrades job, a PYTHONPATH someone exported in a shell profile. The application was correct; the substrate underneath it was not reproducible.

A container image inverts the ownership. Application + runtime + dependencies = container image, built once, addressed by content, and never mutated afterwards. The host contributes a kernel and a container runtime and nothing else that your application can see. Two hosts running the same image are running the same userland, because the userland shipped inside the artifact.

That is the whole operational proposition, and it is narrower than the marketing. Containers do not make a workload scalable, resilient or cheap. They make it reproducible and portable, which is the precondition for the automation — rollout, rollback, autoscaling, scheduling — that the Why Orchestration Exists and The Pipeline as Infrastructure lessons build on top.

What each layer contributes to a running container, and how a failure there presents itself.
Application code
provides The business logic and its entrypoint command.
fails as Exits non-zero on start; the runtime reports the container as exited with a code, not as "crashed".
Language + system dependencies
provides Interpreters, shared libraries, CA certificates, timezone data.
fails as A missing .so or an empty CA bundle — TLS calls fail everywhere at once with a certificate-verify error.
Base image userland
provides A minimal filesystem: shell, libc, package manager, /etc.
fails as A base rebuilt with new package versions changes behaviour on a build that touched no application code.
Container runtime
provides Unpacks layers, applies isolation, sets the process up and supervises it.
fails as Image pull errors, permission errors on volumes, OOM kills delivered as a terminated container.
Host kerneldepth: Operating Systems — namespaces, cgroups, process isolation
provides Namespaces, cgroups, the syscall interface — one kernel shared by every container on the host.
fails as A kernel-level problem is a *host-wide* problem: every container on that machine is affected at once.
Host machine (VM or metal)
provides CPU, memory, local disk, a network interface.
fails as Instance loss takes every container on it; this is why Failure Domains is a scheduling concern, not a container concern.

Image → runtime → running container

Three nouns get used interchangeably and they are not the same thing. An image is a stack of read-only filesystem layers plus a metadata document that names the entrypoint, environment defaults, exposed ports and working directory. A container runtime is the daemon or shim that turns that image into a process. A running container is a process tree that sees the image's filesystem as its root, plus one thin writable layer on top that exists until the container is deleted.

The direction matters when you debug. An image cannot be "restarted" and a container cannot be "edited" in any way that survives — anything written into that top writable layer disappears with the container, which is the entire premise of Persistent Data and Containers. When someone says "I fixed it on the box", in a container world they have fixed exactly one replica until its next restart.

One image also produces many containers. That is what makes horizontal scaling cheap: starting the eleventh replica costs a process start, not a machine build, provided the image is already on the host. When it is not, you pay the pull — see Why Image Size Is an Infrastructure Problem.

From a stored artifact to a supervised process
pull by digestunpack + startsame image, second processsyscallssyscallsRegistry (image + digest)Container runtime on the hostContainer 1 (writable layer)Container 2 (writable layer)Host kernel (shared)
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

What this domain teaches, and where the internals live

containers· Linux containers. Windows containers use different kernel primitives and cannot run Linux images natively.

A container is not a lightweight virtual machine and it is not a new kind of process. It is an ordinary Linux process whose view of the system has been narrowed by kernel features that existed before containers were a product. Those features — namespaces for what the process can *see*, cgroups for what it can *consume* — are taught properly in the Operating Systems domain, and this module deliberately does not repeat them.

What is specific to infrastructure is everything downstream of that: the artifact, its build pipeline, the registry it is promoted through, the configuration injected at start, and the fact that the isolation boundary is thinner than a hypervisor's. That last point is a real constraint, not a footnote — see Containers vs Virtual Machines.

MechanismWhat this module says about itWhere the depth lives
Namespaces (PID, mount, net, UTS)The reason a container sees its own filesystem and process table.Operating Systems → process isolation
cgroupsThe reason a CPU or memory limit is enforceable, and the reason an OOM kill looks like a vanished container.Operating Systems → resource limits
Union/overlay filesystemWhy layers are shared between images and why the writable layer is disposable.Operating Systems → file systems
Container networkingThat each container gets an interface and that ports must be published to be reachable.Networking → container networking
Image format and registry protocolTaught here: layers, digests, tags, promotion.This module — What Is Inside a Container Image, The Container Registry
Scheduling containers across hostsExplicitly not a container feature. It is what an orchestrator adds.Why Orchestration Exists
The division of labour between domains. Follow the right-hand column for depth.

Key points

  • Application + runtime + dependencies = image. The host supplies a kernel and a runtime, and nothing else your application can see.
  • Containers buy reproducibility and portability. They do not buy scalability, resilience or lower cost — those come from what you build on top.
  • Image, runtime and running container are three different nouns; only the image is durable, and only the image is what you promote.
  • One image starts many containers, which is why horizontal scaling becomes a process start rather than a machine build.
  • The isolation is kernel features applied to an ordinary process. The internals are taught in Operating Systems; this domain teaches the operational consequences.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • A build produces an image: ordered read-only layers plus a config document with entrypoint, env defaults, user and working directory.
  • The image is pushed to a registry and referenced by tag or, correctly, by content digest.
  • A host's container runtime pulls the layers it does not already have, unpacks them into an overlay filesystem and adds one writable layer on top.
  • The runtime creates namespaces and cgroups for the process, then executes the entrypoint as PID 1 inside them.
  • The process makes ordinary syscalls to the host kernel — there is no guest kernel and no instruction translation in the path.
  • On stop, the writable layer is discarded. The image is untouched, which is why the next start is identical to this one.
What you still own
  • You still own the base image's patch level. A container does not update itself; a CVE in the base is fixed by rebuilding and redeploying, not by apt upgrade on a host.
  • You own PID 1 behaviour: signal handling, zombie reaping, and shutting down cleanly on SIGTERM — see Graceful Shutdown: The 502 Spike Nobody Investigates.
  • You own resource requests and limits. Unlimited containers are the reason one noisy workload starves its neighbours on a shared host.
  • You own where logs go. A container that writes to a file inside itself has written to a disposable filesystem.
  • You own the host fleet underneath unless a managed container platform runs it for you — containers moved the packaging problem, not the capacity problem.
How it fails
  • The image builds and runs on an ARM laptop and fails to start on an x86 host with an exec-format error — architecture is part of the artifact.
  • The container exits immediately with code 0 because the entrypoint was a command that finishes; the platform restarts it in a loop and reports a crash loop.
  • The application writes uploads to the container filesystem; a routine restart deletes them and no alert fires because no component failed.
  • A memory limit is hit and the kernel kills the process. From the application's side there is no exception and no log line — the process simply ends.
  • The image runs as root because nobody set a user, so a container escape or a mounted host path becomes a host-level compromise.
How it scales
  • Startup is a process start plus, on a cold host, an image pull. The pull is usually the dominant term and is what makes scale-out lag — see Why Image Size Is an Infrastructure Problem.
  • Density is bounded by memory before CPU on most services: every replica carries its own runtime heap even though the image layers are shared on disk.
  • Layer sharing means ten containers from one image cost roughly one image on disk, but ten images from ten unrelated bases cost ten.
  • What runs out first on a single host is usually file descriptors, PIDs or ephemeral ports long before the CPU is saturated.
Security
  • The trust boundary is the host kernel, shared by every container on the machine. It is a real boundary and a thinner one than a hypervisor's.
  • Default-root is the most common avoidable finding: set a non-root user, drop capabilities, and make the root filesystem read-only where the workload allows it.
  • The image is a supply-chain artifact — every layer beneath your code was written by someone else. See The Infrastructure Supply Chain and The Container Registry.
  • A container is not an authorization boundary. It does not decide what your workload may call; identity and policy do — see Human vs Workload Identity.
Cost shape
  • Containers themselves are not a line item. What they change is utilization: several workloads packed onto one instance instead of one VM each.
  • The bill moves to registry storage, image transfer through the egress or NAT meter, and the build minutes that produce the images.
  • Density improves cost only if requests and limits are set honestly. Over-requested containers reserve capacity nobody uses — see Idle Capacity: Headroom or Waste?.
What to watch
  • Container restart count and last exit code — the fastest way to distinguish a crash from an OOM kill from a failed health check.
  • Image pull duration and pull failures, which explain scale-out lag and most "the deploy is stuck" reports.
  • Per-container CPU throttling and memory working set against the configured limits, not against the host's totals.
  • The signal that lies: host-level CPU and memory. A host at 40% can still be throttling a single container into timeouts, because the limit is per-cgroup.
Simpler alternatives
  • A single self-contained binary — a Go executable or a fat JAR — copied onto a VM by the deploy pipeline. If dependency drift is your only problem, this solves it with no runtime, no registry and no image build.
  • An immutable machine image built by the same pipeline. Same reproducibility argument, coarser unit, and it fits teams already running VM fleets — see Mutable Servers and Immutable Images.
  • A platform-as-a-service that builds from source. You get containerization without owning Dockerfiles, at the cost of the platform's opinions.
  • For a single service on a single VM with a stable OS, plain processes under the system service manager remain a defensible and much simpler answer.
What adopting this costs
  • Buys reproducibility across environments; costs a build pipeline, a registry, an image lifecycle policy and a new class of failure (pull, layer, entrypoint).
  • Buys density; costs the discipline of setting limits, because unbounded containers on a shared host make each other unpredictable.
  • Buys portability of the artifact; does not buy portability of everything around it — load balancers, secrets, storage and identity remain provider-shaped.
  • The isolation boundary is thinner than a VM's. For hostile multi-tenant code that is a genuine disqualifier, not a preference.

What people believe, and what is true

Claim

Containers are lightweight virtual machines.

Reality

There is no guest kernel and no virtual hardware. It is a normal process with a restricted view, which is why it starts in milliseconds and why the isolation is weaker.

Claim

Containerizing an application makes it scalable.

Reality

It makes the artifact reproducible. A stateful, single-writer application in a container is still a stateful, single-writer application.

Claim

If it runs in the container it will run anywhere.

Reality

Anywhere with the same CPU architecture, a compatible kernel, and the same external dependencies. The image carries userland, not the world around it.

Apply it