Buildsimplementation

Hermetic Compilation

A build is hermetic when its result depends only on its declared inputs — not on which `cc` happens to be first in `PATH`, not on a header that exists on one laptop, not on anything fetched from the network while it runs.

The question

Why does this build work on my machine and fail in CI, when the commit is identical?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

A build action as a closed function: a declared input set — sources, headers, the toolchain binary itself, the sysroot, the command line, the target description — mapped to an output set. Nothing else is in scope. The representation exists to answer one question the source tree cannot: *is this the complete list of things that can change the answer?* A Makefile rule is not that representation, because it names the inputs its author remembered.

What this phase may assume or do

A build action may be treated as a pure function of its declared inputs only if the process that runs it can read nothing outside them: no ambient PATH lookup, no implicit /usr/include, no environment variable consulted by a script, no network fetch, no clock, no hostname, no user name, no /dev/urandom. Under that condition the input hash is a sound cache key and two machines that agree on the inputs must agree on the outputs. When any undeclared read exists, the same key can map to two different artifacts, and every cache, every incremental decision and every reproducibility claim built on that key is unsound.

Key points

  • Hermetic means the build can read only its declared inputs. Reproducible means running it twice yields identical bytes. They are different properties and each fails without the other.
  • The common leaks are ordinary conveniences: PATH lookup, default include paths, environment variables, network fetches, an unhashed compiler, absolute paths, the clock and nondeterministic parallelism.
  • Enforcement is by sandbox, not by discipline: only declared inputs are mounted, so an undeclared read fails at development time instead of corrupting an artifact later.
  • Hermeticity is the precondition that makes a content-addressed cache key sound, which is what makes shared and remote caching safe rather than hopeful.
  • The cost is mostly human — enumerating inputs, and a new class of loud failure where the build previously worked by accident.
  • Its most underrated value is diagnostic: when the input set is finite and printable, a build mismatch is a diff rather than a search of two whole machines.

Hermetic is about inputs; reproducible is about outputs

These two words get used interchangeably and they are not the same property. Hermetic says the build cannot see anything you did not declare. Reproducible says that running it twice produces byte-identical artifacts. Hermeticity is a property of the *environment*; reproducibility is a property of the *toolchain running inside it*.

The distinction matters because each can fail without the other. A perfectly sandboxed build of a compiler that stamps __DATE__ into the binary, or that iterates a hash map whose ordering varies with an address, is hermetic and not reproducible — nothing undeclared came in, and the output still differs. Conversely a build that reads /usr/include/stdio.h off the host is not hermetic, yet it will happily produce identical output all year on machines that happen to have the same libc. That second case is the dangerous one, because it looks fine right up until someone upgrades a base image.

The practical relationship is one-directional: hermeticity is what makes reproducibility *achievable* and, more importantly, what makes a failure to reproduce *diagnosable*. Without it, a mismatch has an unbounded search space — anything on either machine could be the cause. With it, the input set is finite and printable, so the difference is in a list you can diff. That is why [[reproducible-compilation]] is usually pursued by first closing the environment and only then hunting the nondeterminism inside the compiler.

Two properties, four combinations, all of which occurtypical
ReproducibleNot reproducible
HermeticThe goal. A key implies an artifact, and remote caching is sound.Sandbox is closed but the compiler embeds a timestamp, an absolute path or a hash-ordered iteration. Diagnosable, because the input set is finite.
Not hermeticThe comfortable illusion. Works while every machine happens to agree, and breaks silently on the first base-image bump.The default state of an unmanaged build. A mismatch has no bounded search space; this is the "works on my machine" report.

What actually breaks it

implementationWhich of these bite you is toolchain-specific. Clang and GCC both embed absolute paths in DWARF unless -ffile-prefix-map / -fdebug-prefix-map is used, and both honour SOURCE_DATE_EPOCH for __DATE__ and __TIME__ in recent versions. Go's toolchain records module hashes and refuses to build outside them, so its default posture is far closer to hermetic than a typical C project's. Rust records absolute paths in panic messages unless --remap-path-prefix is passed. There is no cross-toolchain flag; each of these is a separate thing to learn and to check.

Almost none of the leaks are exotic. They are the conveniences that make a build easy to write on one machine, and each one is a channel from the host into the artifact.

The two that cause the most confusion are the ones that fail *intermittently*. A network fetch during the build is fine until the registry serves a new patch version, or is down; a mutable tag such as :latest on a base image or a floating version range makes the input set a function of the calendar. Both produce a build that was correct yesterday and is wrong today with no commit in between, which is the hardest failure to attribute because version control shows nothing.

  • Ambient `PATH`. The rule says cc. Which cc that is depends on the shell that launched the build. Two engineers with different Xcode or toolchain installs are compiling with different compilers and neither knows.
  • Undeclared system headers and libraries. #include <stdio.h> resolves through the default include path into the host's libc. Different distributions, different glibc, different struct layouts — and the object file differs in ways that only surface at link time or, worse, at run time.
  • Environment variables. CFLAGS, CPATH, LD_LIBRARY_PATH, LANG, TZ, SOURCE_DATE_EPOCH, NODE_ENV, and whatever a build script decided to read. Locale in particular changes sort order, which changes link order, which changes symbol layout.
  • Network access during the build. Fetching a dependency, a schema or a base image while compiling makes the artifact a function of a remote server's state at that instant.
  • The toolchain itself, when it is not an input. If the compiler binary is not hashed into the key, a patch-level upgrade silently changes every output while every key stays the same.
  • Absolute paths. Debug info, __FILE__, assertion messages and .debug_str all record where the source lived, so building in /home/ana/proj and /home/bo/proj yields different bytes.
  • The clock, the hostname, the user, the build number. __DATE__, __TIME__, an embedded version banner, a build-machine label baked into a binary.
  • Nondeterministic parallelism. Anything whose output depends on which worker finished first: a code generator writing files in completion order, a linker fed inputs by a glob.

How it is enforced, and what enforcement costs

Hermeticity is not something you assert; it is something a mechanism denies you the ability to break. Bazel and Buck2 run each action in a sandbox where only the declared inputs are visible, so an undeclared read fails with "no such file" during development rather than producing a wrong artifact in CI. Nix goes further and builds in a private mount namespace with no network, an empty environment and a fixed fake home directory, with every input addressed by content hash. Container-based builds approximate this by pinning the image by digest and passing --network=none.

The cost is real and it is mostly paid by humans. Every input must be enumerated, which turns "add an #include" into "add an #include and a build-rule edit". A first-time hermeticization of an existing project is measured in weeks, and the work is unglamorous: chasing down each undeclared read the sandbox surfaces. The second cost is a class of hard error you did not have before — a build that used to work by accident now fails loudly. That is the feature, and it is still a change in day-to-day friction that teams underestimate.

The payoff is that the cache key from [[build-system-interface]] becomes trustworthy. Once nothing undeclared can influence an output, an equal key really does imply an equal artifact, so a remote cache and remote execution become sound rather than hopeful — and a CI machine, a laptop and a colleague's container all agree.

The same compile, opened and closed
Before
# ambient
cc -c src/a.c -o a.o
#   `cc` found via PATH
#   <stdio.h> found via the default include path
#   CFLAGS read from the environment
#   version.h generated by a script that calls date(1)
After
# closed
/toolchains/clang-17.0.6-x86_64/bin/clang \
  -nostdinc --sysroot=/sysroots/linux-glibc-2.35 \
  -isystem /sysroots/linux-glibc-2.35/usr/include \
  -ffile-prefix-map=$PWD=. \
  -c src/a.c -o a.o
#   inputs: src/a.c, the sysroot tree, the clang tree, this command line
#   env: cleared; network: none; version.h: a declared generated input
Legal only when

The closed form may be treated as a pure function of its inputs only if the sandbox actually prevents every other read — the compiler binary is inside the input set, the sysroot is complete, the environment is cleared rather than merely unused, and no step in the graph reaches the network. Under those conditions the input hash is a sound cache key.

Illegal when

Any of it is aspirational. A -isystem path that also falls back to the host include path, a compiler wrapper script that consults PATH, a generated header produced outside the graph, or a sandbox with network access left on. Then the key is incomplete and a matching key does not imply a matching artifact — the failure mode is a cache that serves a stale or foreign object with total confidence.

How it works

The steps, in the order the compiler takes them.

  • Every action declares its inputs explicitly, including the toolchain binaries and the sysroot, not just the project sources.
  • The build system materialises exactly those inputs into a fresh directory tree, usually via symlinks or a content-addressed store.
  • The action runs in a sandbox — a mount namespace, a chroot, or a container — where nothing outside that tree is visible and the network is unavailable.
  • The environment is emptied and repopulated with a fixed set of variables, so a variable the host had set cannot reach the process.
  • Path-remapping flags rewrite the build directory to a fixed string, so the artifact does not record where it was built.
  • Any read outside the declared set fails as a missing file, and the build reports it as a missing dependency rather than silently succeeding.
  • The declared input set is hashed into the cache key, so an equal key now genuinely implies an equal output.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • The build passes locally and fails in CI on the same commit, with an error about a missing header nobody has ever seen — the laptop had it installed system-wide.
  • A shared cache serves an object compiled by a different patch version of the compiler, and the link fails with an unresolved symbol that exists in the source.
  • Two engineers get binaries of different sizes from the same commit, and the diff turns out to be their home directory paths in the debug info.
  • A build that succeeded on Friday fails on Monday with no commits in between, because a floating dependency version or a mutable image tag moved.
  • Tests pass in the build sandbox and the shipped binary crashes on a customer machine, because the build linked against the host's newer libc symbols.
  • Enabling the sandbox turns a green build red in fifty places at once, and the team concludes hermeticity is broken rather than that the build always was.

When it helps

  • Any project where more than one machine builds the same code — which is any project with CI, and every project with more than one engineer.
  • Shared or remote build caches, where the entire value proposition rests on a key that genuinely determines the artifact.
  • Cross-compilation and multi-target builds, where "whatever the host has" is not even the right thing to link against — see [[cross-compilation]].
  • Security and audit work, where you must be able to say what went into a binary; an undeclared input is an unauditable input, and this is the ground floor of [[toolchain-trust]].
  • Long-lived software that must still build in five years, when the host distribution no longer has any of the libraries the build silently assumed.

When it hurts

  • Small projects and rapid prototyping, where the full sandbox setup costs more than the class of bug it prevents and a plain make is the honest answer.
  • Builds that legitimately need a device or a service — GPU compilation against a vendor driver, code signing against a hardware token, a licence server — where the pragmatic answer is a narrow, documented, deliberately declared escape hatch rather than an argument about purity.
  • Teams adopting it mid-project without budget for the migration, where the sandbox surfaces dozens of pre-existing undeclared reads at once and the work stalls.

What it costs

Every one of these is paid by something.

  • Sandboxing buys a sound cache key and pays in setup: input materialisation, namespace creation and teardown per action, which is a fixed per-action overhead that hurts most when actions are small.
  • Enumerating every input buys diagnosability and pays in build-rule verbosity, and in ongoing friction — every new dependency is now two edits, not one.
  • Pinning the toolchain into the input set buys correctness across machines and pays in repository size and in upgrade ceremony: bumping the compiler now invalidates every cached artifact at once.
  • Refusing network access at build time buys determinism and pays by forcing a vendoring or lockfile-plus-mirror step, which is real infrastructure someone has to run.
  • Loud failure on an undeclared read buys early detection and pays in a period where the build breaks more often than it used to, on problems that were always there.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Container-based builds pinned by image digest: far cheaper to adopt, and they close the biggest hole — the host toolchain — without per-action sandboxing. They do not stop an action from reading a file another action wrote out of order, so incrementality inside the container is still unverified.
  • Nix and Guix take the strongest position: every input, including the compiler, is content-addressed, and the build runs with no network and a scrubbed environment. Excellent guarantees, a genuinely steep learning curve, and a packaging burden for anything not already packaged.
  • Convention plus CI enforcement — a clean-machine build in CI that would fail on any host leakage. Cheap, catches most leaks eventually, and gives no per-action guarantee, so caching remains untrustworthy.
  • Doing nothing and rebuilding from scratch every time is a legitimate choice for a small project: it removes the incremental-correctness question entirely by removing incrementality.

See it for yourself

The flag, dump or tool that shows you this directly.

  • bazel build --spawn_strategy=sandboxed fails on undeclared reads; bazel aquery --output=text //target prints the declared input set for each action.
  • strace -f -e trace=openat,execve -o build.log <build command> shows every file the build actually opened. Grep the log for paths outside your source tree — that list *is* your undeclared input set.
  • nix-build runs with no network and an empty environment by default; nix derivation show prints the complete input closure of an artifact.
  • gcc -H and clang -H print every header included, with nesting, so you can see which resolved to a system path.
  • Build the same commit in two directories with different names. Any byte difference in the artifacts is a path leak — usually debug info, fixed by -ffile-prefix-map.
  • diffoscope a.o b.o explains a binary difference in structured terms rather than as a hex dump, which is what makes the last mile of this work tractable.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "We build in Docker, so the build is hermetic." A container pins the base filesystem and nothing else. Network access, environment variables, mutable tags and host mounts all still reach inside.
  • "It builds the same everywhere, so it must be hermetic." It builds the same everywhere *so far*. Undeclared inputs that happen to agree today are the failure that is waiting, not the absence of one.
  • "Hermetic and reproducible mean the same thing." One is about what the build can read; the other is about what it writes. A hermetic build of a compiler that embeds a timestamp is still not reproducible.
  • "The compiler is part of the environment, not an input." It is the single most important input. A compiler upgrade that does not invalidate the cache is exactly how a stale object survives into a release.

Misconceptions

The claim, and what is actually true.

Hermeticity is a purity concern for large companies.
It is the precondition for a trustworthy cache key. Any team sharing build artifacts between machines is relying on it whether or not they have enforced it.
If the sandbox breaks my build, hermeticity broke it.
The sandbox reported a dependency that was always missing from the declaration. The build was relying on a file it never asked for.
Pinning versions in a lockfile makes a build hermetic.
It closes one channel — dependency resolution. The compiler, the system headers, the environment and the clock are all still open.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

A hermetic build can only see the things it declared it needed. It cannot pick up whichever compiler is first in your PATH, cannot quietly use a header that exists on your laptop, and cannot download anything while it runs. That is what makes the same commit produce the same result on your machine and in CI.

practical

You do not have to adopt Bazel to get most of the value. Pin the compiler and reference it by an absolute path from a checked-in or downloaded toolchain. Pass an explicit sysroot instead of relying on default include paths. Clear the environment rather than assuming nothing reads it. Add -ffile-prefix-map so debug info does not record your home directory. Turn off network access during the build and vendor or mirror what you need. Then run strace -f -e trace=openat over a build and read the list of files it opened outside your tree — that is the remaining work, and it is usually shorter than you expect.

advanced

The interesting question is where the boundary of the input set legitimately sits, because "everything" is not achievable. The kernel is an input to every build and nobody hashes it. /proc layout, the CPU model that a -march=native build reads, the page size, the filesystem case-sensitivity: these are real influences that most systems declare out of scope by fiat. Nix draws the line at the kernel ABI, Bazel at the sandbox mount tree, containers at the image digest. Each line is defensible and none is complete, so hermeticity in practice is not a boolean but a statement of *which* channels have been closed — which is why an honest build system publishes its assumptions rather than claiming purity. That framing is also what stops the pursuit from becoming absurd, and it is the same reasoning that stops [[trusting-trust]] from being a counsel of despair: you cannot close every channel, so you name the ones you have closed and the ones you are trusting.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationHow close a toolchain gets to hermetic by default varies enormously. Go and Rust ship lockfiles and content-addressed module caches and are close out of the box; C and C++ builds resolve headers through ambient include paths and are the furthest away. What breaks hermeticity in one ecosystem is often not even expressible in another.
typicalMainstream sandboxing on Linux uses mount and network namespaces, which are cheap. On macOS, Bazel uses sandbox-exec, which is slower and less complete, and on Windows sandboxing is weaker still — so the same build rules give different strength guarantees per platform, and CI is usually the strictest of the three.
targetThe sysroot, target triple and ABI must be inside the declared inputs, not inherited from the host. A build that is hermetic with respect to sources but takes its libc headers from the build machine will produce objects that link on that machine and fail on the deployment target.

If you were asked this in an interview

  • What is the difference between a hermetic build and a reproducible one, and can you have either without the other?
  • Your build passes locally and fails in CI on the same commit. What are the first four things you check?
  • What has to be in the input set for a remote build cache to be sound, and what is the consequence of omitting the compiler binary?

Connections

DSAhash-table
Securitysupply-chain-securitydependency-securityartifact-integritysandboxing
Cloudcontainer-imagescicd-infrastructureenvironment-managementinfrastructure-supply-chain
OS & Networkingprocess-isolationprocess-creationfiles-and-paths
Domains that do not exist yet
  • DevOps / Production Engineering — Build infrastructure: sandboxed remote execution, artifact registries, base-image policy and mirror operation
    Running the sandbox fleet, deciding who may publish to a cache, and operating the mirrors that replace network access during a build are all that domain's work. We stop at the compiler-side question — which reads can influence an object file, and what has to be declared for a key to mean anything.