Build Environments
The machine a build runs on is an input to the build, and everything about it that is not declared is a source of drift, of irreproducibility and of shared-state compromise.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What about the machine running this build is an input, and which of those inputs have we actually declared?
Builds pick up whatever the environment offers — a compiler on the PATH, a system library, an environment variable, network access — and none of it appears in the commit that the build is supposedly built from.
Install the toolchain on the build server once, keep it updated, and every build uses it. It is fast because nothing has to be set up per build.
The server becomes an unversioned dependency of every build. Nobody can say what is installed on it, and the person who set it up has left (Pets and Cattle, Read Carefully).
- The server becomes an unversioned dependency of every build. Nobody can say what is installed on it, and the person who set it up has left (Pets and Cattle, Read Carefully).
- Updating it changes every build at once, with no commit and no review, and the failures are attributed to whatever change happened to land that day.
- State persists between jobs. A file written by one build, a tool installed by hand, a poisoned cache, a background process — the next build inherits all of it.
- For fork pull requests that is a security boundary failure, not just a hygiene one: untrusted code runs on a machine that trusted builds will run on next (CI Security).
- "Works on the build server" and "works on a developer laptop" diverge, and the difference is invisible because neither environment is described anywhere (Environment Drift).
- Disk fills, and a build fails for reasons that have nothing to do with any change (Triaging a CI Failure).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A build has two input sets: the ones it declares and the ones it absorbs. Every property of the environment is in one set or the other, and the second set is the problem.
- Hermeticity is the degree to which the second set is empty. It is a spectrum, not a switch, and the useful moves along it are ordered: container the build, pin the image by digest, remove network access, pin the toolchain as a declared dependency.
- Containerising a build converts "whatever is installed" into "this image digest", which is a declared, versioned, reviewable input. That single step removes most environment drift (Containers Are Processes With the Kernel’s View Narrowed).
- It does not remove network access. A container that runs
apt-get installor downloads a script at build time is picking up undeclared inputs from inside a nicely versioned box. - Ephemerality addresses a different axis: not what a build can reach, but what it can leave behind. A runner destroyed after each job cannot pass state to the next one.
- Those two axes are independent, and confusing them is common — an ephemeral runner with full network access and no image pinning is isolated in time and not in inputs.
Everything the build absorbs
Each row is something a build reads that is usually not written down anywhere. The remedy column is the work: converting an absorbed input into a declared one.
Work top to bottom — the first four rows account for most reported "it works on my machine" build failures.
| Ambient input | How it gets in | Symptom when it drifts | Declare it by |
|---|---|---|---|
| Toolchain version | Whatever is on PATH | Different binaries, sometimes different behaviour | Pin the compiler and runtime as declared dependencies |
| OS and system libraries | The host or the base image | Link errors, or a binary that runs only on the build host | Pin the build image by digest (Tags Versus Digests) |
| Network reachability | Any fetch during the build | Builds break when a registry is slow or a package is yanked | Vendor or mirror; disable the network for the build step |
| Environment variables | Runner configuration and shell profile | Different behaviour with no code change | Enumerate what the build reads; fail loudly on absent (Validate at Startup, Fail Clearly) |
| Filesystem paths | Where the checkout happens to be | Debug info and some binaries differ between machines | Build at a fixed canonical path, or use path remapping |
| Clock and timezone | Host settings | Time-dependent tests and embedded timestamps vary | Pin TZ; set SOURCE_DATE_EPOCH (Production Time Is UTC) |
| Locale | Host settings | Sort order and formatted output differ | Pin LC_ALL |
| CPU count and memory | Runner size | Timeouts, OOM kills, different parallel scheduling | Set explicit limits rather than reading what the host offers (Requests and Limits) |
| Leftover state | The previous job on a persistent runner | Builds that pass only on one runner | Ephemeral runners; clean workspace per job |
Hermeticity is a spectrum
These are ordered by how much of the ambient set they eliminate, and each step costs more than the one before. Most teams should be at the second or third; very few need the fourth.
The useful discipline is knowing which step you are on, because each one has a specific set of things it does not fix.
How much of the machine is allowed to be an input?
when Getting started, or builds that genuinely need host access. Fastest, because nothing is set up per job.
cost Every property of the host is an undeclared input, and the host changes without a commit. Do not use for anything that publishes (CI Security).
when The default for almost everything. Converts the environment into a versioned, reviewable artefact.
cost Image maintenance and periodic bumping. Does not remove network access, so downloads during the build remain undeclared.
when You want reproducibility you can actually verify, and dependency availability you control.
cost Needs vendoring or a mirror, and a pre-fetch step whose own inputs must be pinned (Dependency Pinning).
when Large repositories where cross-machine reproducibility and remote caching are load-bearing.
cost Whole-repository adoption. Every tool must be declared, and the escape hatches teams add under deadline pressure remove the guarantee (What a Build System Actually Is).
What persistence lets through
Ephemerality is a separate question from hermeticity, and the failures below are all about what one job leaves for the next one. On a shared persistent runner they range from confusing to a full compromise.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Previous job wrote to the workspace | Build passes on one runner and fails on another | Stale files treated as build inputs | Clean checkout per job; ephemeral runners make it structural rather than a setting |
| Someone installed a tool by hand months ago | A new runner cannot build at all | The runner's configuration exists only as its current state (Pets and Cattle, Read Carefully) | Rebuild runners from an image definition; treat manual installation as an incident |
| Fork pull request ran on a shared runner | Later trusted build behaves unexpectedly | Untrusted code left a modified tool, a poisoned cache or a background process | Untrusted triggers never touch persistent runners; destroy the runner after each job (CI Security) |
| Disk fills over weeks | Unrelated builds fail on write errors | Nothing cleans up artefacts, images and caches | Ephemeral runners, or enforced cleanup with monitored free space (Capacity Management) |
| Provider updates the hosted runner image | Every build changes behaviour on the same day, no code change | The runner image is an unpinned dependency | Pin the build container by digest so the host image matters less; subscribe to the provider's change announcements (Change Correlation) |
| Build reads a credential from the environment | Works in CI, fails locally, or leaks into a log | An ambient secret rather than a declared, scoped one | Inject secrets per job at the narrowest scope; never in a job that runs untrusted code (Secrets in CI) |
How to do it properly
Most important first.
- Run builds in a container image pinned by digest, and treat the image definition as source under review (Tags Versus Digests).
- Pin the toolchain explicitly — compiler, runtime, package manager — rather than relying on what the image happens to ship.
- Prefer ephemeral runners. A fresh environment per job removes an entire class of "why did it work yesterday".
- Remove network access from the build step where you can, or restrict egress to your own mirror. A build that cannot reach the internet cannot absorb an undeclared input from it (Egress Security).
- Never run untrusted code on a persistent shared runner. If self-hosted runners are required, exclude fork pull requests and destroy the runner between jobs (Sandboxing Untrusted Workloads).
- Declare environment variables the build reads, and fail if one is missing rather than defaulting silently (Validate at Startup, Fail Clearly).
- Keep a scheduled cold, clean build as the detector: it is what tells you the environment has started contributing inputs (Caching in CI).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A compromised or drifted build environment affects every artefact built on it. Containment is ephemerality, image pinning, and provenance records that let you bound the affected window.
What can go wrong
- A self-hosted runner nobody can rebuild, whose configuration exists only as its current state.
- A runner image updated by the provider, changing every build with no commit — and no way to correlate the failures until someone thinks of it.
- Leftover state producing a build that passes only on the runner that ran the previous job.
- A container build that installs OS packages at build time, so the same Dockerfile produces different images as upstream repositories move (Reproducible Builds).
- Ephemeral runners without a warm cache, making every build cold and slow, so someone re-introduces persistence to fix the speed and re-introduces the state problem with it.
- Build credentials left in the environment of a job that runs untrusted code (Secrets in CI).
- Resource limits so different from a developer machine that tests pass locally and are killed in CI, or the reverse (OOMKilled: Over the Memory Limit).
- "We build in Docker, so the build is hermetic." Containerised is not hermetic. If the build downloads anything, the network is still an input (Reproducible Builds).
- "Ephemeral runners make CI secure." They remove persistence between jobs. They do nothing about what a job can reach while it runs.
- "The runner is managed, so the environment is stable." Managed runner images are updated by the provider on their schedule, and that is a change to your build inputs.
- "It only fails in CI, so it is a CI problem." CI is usually more constrained and less warm than a laptop. A failure there is often the first honest test the code has had (Why Local Success Predicts So Little).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The build image is referenced by digest, and that digest appears in the provenance record (Build Provenance).
- A build on a brand-new runner, with an empty cache, succeeds — scheduled, and its failure treated as a real defect.
- A build with networking disabled either succeeds, or fails naming exactly which undeclared input it wanted.
- You can recreate the build environment from source: a Dockerfile or an image definition in the repository, not a machine someone configured.
- No job that runs untrusted code shares a runner with a job that holds credentials — verified from the pipeline configuration, not from intent.
- Pinning the build image by digest means rolling the environment back is a one-line revert, which is most of the argument for doing it.
- When a build breaks after an environment change, roll the environment back first and investigate afterwards — it restores everyone else's ability to work (Triaging a CI Failure).
- A compromised persistent runner cannot be rolled back by cleaning it. Destroy it, rebuild from the image definition, and treat artefacts built on it as suspect (Build Provenance).
- Automate runner provisioning from an image definition, so a runner is reproducible rather than maintained.
- Automate destruction after each job; the cheapest isolation available is not reusing the machine.
- Automate detection of undeclared inputs with the scheduled network-disabled build.
- Do not automate away the human decision about which triggers may run on which runner pool. That is a trust boundary (Trust Boundaries).
- Ephemeral runners cost cold starts on every job, which you buy back with a well-keyed remote cache — and a shared cache is its own trust boundary (Caching in CI).
- Hermetic builds are the strongest guarantee and the most restrictive to work inside; every tool has to be declared, including the ones someone needs for a one-off diagnosis.
- Self-hosted runners buy specific hardware, private network access and cost control, and hand you the operation and the isolation problem.
- Pinning the runner image by digest stops drift and means the pin needs deliberate updating, or you are building on an image that ages (Dependency Pinning).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGitHub-hosted runners are fresh VMs per job with a preinstalled image the provider updates on a published schedule; GitLab runners can be shared, group or project scoped with executor types ranging from Docker to shell (where "shell" means no isolation at all); Jenkins agents are whatever you built. The isolation you get by default differs from none to a complete VM.
- PLATFORM-SPECIFICContainer-based isolation depends on the host kernel, so a Linux container build cannot produce Linux binaries identical to a macOS-hosted one without a Linux VM underneath. Cross-compilation and emulation each introduce their own differences, which is why "same Dockerfile" does not imply "same artefact" across architectures.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.