The Builder Is Inside the Trust Boundary
Every downstream control attests to whatever the builder produced — so if the build environment can be influenced, signatures, SBOMs and provenance all faithfully describe a compromised artifact.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does it matter where the artifact was built, if we sign it and verify the signature?
Signing, provenance and inventory all describe the output of the build. None of them can tell you whether the build itself produced what the source said it should.
Any runner that can produce the artifact is fine. The build is deterministic enough, and the controls after it — signature, SBOM, scan — will catch anything wrong.
Those controls describe the artifact, not its derivation. A compromised build produces a correctly signed artifact with an accurate SBOM of its own contents.
- Those controls describe the artifact, not its derivation. A compromised build produces a correctly signed artifact with an accurate SBOM of its own contents.
- Provenance generated by a step inside the build is a claim the build makes about itself. If the build can be influenced, so can the claim.
- A shared, long-lived runner carries state between builds: caches, checkouts, tool installations, and sometimes credentials. Yesterday's job is part of today's build environment (Build Environments).
- A builder with interactive access has an unaudited path to modify the output that leaves no trace in the source repository.
- If the build definition can be supplied by the change being built, then proposing a change is equivalent to modifying the build.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The trust boundary includes anything that can influence the bytes. That means the runner image, the build definition, the tooling, the network the build can reach, and any state that survives between builds.
- A trusted builder has four properties: the build definition comes from the same reviewed source as the code; the environment is ephemeral and isolated; there is no interactive access; and the signing identity is available to the builder, not to the build steps.
- The fourth is the one that is usually missing. If a build step can read the signing credential, then anything that runs in the build — including a dependency's install script — can sign arbitrary content as you (Securing the Pipeline Itself).
- Provenance must be produced out of band: recorded by the platform running the build, describing what it observed — which source, which definition, which builder identity — rather than assembled by a step the build controls.
- Reproducibility is the independent check. If two builders can produce byte-identical output from the same source, a discrepancy is detectable without trusting either of them (Reproducible Builds).
- Published frameworks describe this as levels of build integrity, increasing from "provenance exists" through "provenance is generated by the platform" to "the build runs in an isolated environment the definition cannot escape". Naming the level you are at is more useful than claiming the framework.
Where the trust boundary actually sits
Draw the boundary around everything that can influence the output bytes, and the shared warm runner immediately looks different from the way it looks on a cost report.
Two ways to produce provenance
This is the distinction that decides whether provenance is evidence or decoration, and it is easy to get wrong because both approaches produce a file that looks the same.
steps:
- build the artifact
- run a script that writes
provenance.json:
source: <whatever the script says>
builder: <whatever the script says>
- sign it with the key in the env
the build attests to itself,
using a credential the build can readplatform records, outside the job: which commit was checked out which definition ran which builder identity executed it which artifact digest resulted platform signs that record with an identity no build step can reach
In the first, everything the provenance says is something the build environment supplied, so a build that has been influenced supplies influenced provenance and signs it correctly. In the second the claim comes from the component observing the build rather than from inside it, which is the only version that survives the build being wrong.
Choosing a builder
There is no universally correct answer here; there is a set of properties and a set of costs. What matters is that the choice is made deliberately and the resulting level is stated.
You need a builder for artifacts that reach production.
when The default for most teams. Fresh environment per job, isolation you did not have to build.
cost No warm caches, so slower builds; and you accept the provider's isolation properties rather than verifying them (Caching in CI).
when The build needs private network access — an internal registry, a licence server, a database.
cost You operate the runner fleet: images, patching, isolation per trust level. It is production infrastructure with production obligations.
when Build times are dominated by warm state and the artifacts do not reach production.
cost State crosses between builds. Acceptable for pull request validation; not for release builds.
when You ship to external consumers, or a regulator asks how an artifact was produced.
cost More constrained build definitions, a migration effort, and a hard dependency on that service being available to release at all.
How to do it properly
Most important first.
- Run builds on ephemeral, single-use environments. Destroying the environment after each build removes an entire class of cross-build influence.
- Take the build definition from the protected branch, not from the change being built, for anything that produces a releasable artifact (Protected Branches).
- Keep the signing identity out of the build steps — signing performed by the platform, or by a separate job that consumes the build output and cannot be influenced by it.
- Restrict what the build can reach on the network: a dependency proxy you control rather than arbitrary egress, so what enters the build is recorded (Dependency Management).
- Generate provenance from the platform, and verify it at admission alongside the signature — the same source repository, the same build definition, the same builder (Signing and Verifying Artifacts).
- Make builds reproducible where the cost is reasonable, and check reproducibility on a schedule rather than only aspiring to it.
- If self-hosted runners are required, treat them as production hosts: no interactive access, immutable images, per-trust-level isolation.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing downstream contains a compromised builder — every subsequent control attests faithfully to its output. Containment is upstream: isolation, ephemerality, and keeping the signing identity out of reach of build steps.
What can go wrong
- Provenance generated by a script in the build, which attests to whatever it is told.
- Persistent runners with warm caches, which is a performance decision that silently moves the trust boundary (Caching in CI).
- The build definition editable by the pull request that triggers it, so proposing a change is proposing a build.
- A signing credential exported into the build environment for convenience.
- A managed builder assumed to be isolated without checking, when the isolation depends on configuration you have not set.
- Reproducibility claimed and never tested, so an unpinned input has been drifting for a year unnoticed.
- Unrestricted network access from the build, so an install script can fetch anything and the SBOM records only what it declared.
- "We use a hosted CI service, so the builder is trusted." The service provides isolation primitives. Whether your configuration uses them — ephemeral runners, restricted permissions, definition from a protected branch — is yours.
- "Provenance proves the artifact is good." It records how the artifact was produced. Its value depends entirely on whether the recorder could be influenced by what it was recording.
- "Reproducible builds are the goal." They are one verification mechanism. An isolated, ephemeral builder with platform-generated provenance provides more practical assurance for most teams, and the two are complementary.
- "Self-hosted runners are insecure." They are a host you now operate. Treated as production infrastructure they are fine; treated as a long-lived shared machine somebody set up once, they are the weakest link in the chain.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Two independent builds of the same commit produce the same digest, checked on a schedule rather than asserted.
- The build environment cannot be reached interactively — verified by trying, not by policy.
- Provenance for a deployed artifact names the expected repository, definition and builder identity, and admission control checks all three.
- The signing identity does not appear in the credential set available to build steps.
- Moving from persistent to ephemeral builders slows builds and breaks anything relying on leftover state. Expect to discover undocumented dependencies on the runner's contents.
- If a builder is suspected compromised, the response is to rebuild every artifact produced since the suspected point on a clean builder, and to rotate every credential the builder could reach. Being able to enumerate that set is the preparation (Rotation That Applications Survive).
- Automate: ephemeral environment provisioning, provenance generation by the platform, verification at admission, and scheduled reproducibility checks.
- Keep human: approving a new builder into the trusted set, and any exception that runs a release build somewhere else. That list is the trust boundary written down (Change Management).
- Ephemeral, isolated builders are slower and more expensive than warm shared ones, and the difference is felt on every build (Caching in CI is where the pressure to regress comes from).
- A managed trusted-builder service removes a large amount of work and adds a dependency whose isolation properties you accept rather than verify.
- Reproducible builds are genuinely hard for some toolchains — timestamps, path embedding, parallel build ordering — and the last few percent can cost more than it returns.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICHosted CI services differ in what isolation they provide by default: some give a fresh virtual machine per job, some reuse containers, and self-hosted runners give you whatever you built. The properties to check are the same everywhere; the defaults are not, and the defaults are what you get if you do not look.
- ORG-SPECIFICHow much build integrity to invest in depends on what an artifact can reach and who consumes it. A team shipping software to external customers has an obligation an internal tool does not, and the honest answer for many internal services is a level below the maximum, stated explicitly rather than implied.
- SIMPLIFIEDPresented as four properties. Published build-integrity frameworks decompose this considerably further and add requirements around retention, distribution and independent verification. The four properties are where the practical risk concentrates, not the complete picture.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — reproducibility as a testable property of a build system, and what it costs to keep it true once it is achieved.