Build Provenance
Six facts recorded at build time — commit, builder, inputs, toolchain, timestamp and artefact digest — that let you answer "where did this artefact come from" without guessing.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Something is running in production. Can we prove which commit it was built from, by what, and from which dependencies?
During an incident and during a security response, the first question is what is actually running. Without provenance recorded at build time, that question is answered by inference from tags, deploy logs and memory — and inference is wrong often enough to matter.
The image tag has the version number and the deploy log has the commit. Between the two we can always work it out.
Tags are mutable. v1.4.2 is a pointer, and a pointer can be moved — by a re-run, by a retag, or by an automated release job. What ran under that tag last week is not necessarily what runs under it now (Tags Versus Digests).
- Tags are mutable.
v1.4.2is a pointer, and a pointer can be moved — by a re-run, by a retag, or by an automated release job. What ran under that tag last week is not necessarily what runs under it now (Tags Versus Digests). - Deploy logs record what was requested, not what was produced. A deploy of
latestrecordslatest. - The commit alone is not enough. Two builds of the same commit can differ if dependencies were unpinned or the toolchain moved (Reproducible Builds).
- Nothing in a bare artefact says which build produced it, so a locally built image pushed by hand is indistinguishable from one built by CI.
- When a dependency is found to be compromised, the question is which artefacts contain it. Without a recorded input set, the honest answer is "we do not know" and the response is to rebuild and redeploy everything (The Delivery Chain as Attack Surface).
- Rebuilding to find out changes the thing you were trying to inspect, and if the build is not reproducible it does not answer the question anyway.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Provenance is a record produced *by the builder*, at build time, binding an artefact identity to the facts about how it came to exist.
- The artefact identity must be a content digest, not a name or a tag. Everything else in the record is anchored to it.
- The six facts are: the source commit, the builder identity, the resolved input set, the toolchain, the build timestamp, and the resulting digest. Each answers a different question, and the missing one is always the one you need.
- Trust in the record comes from who produced it. A provenance statement the build script wrote about itself is a claim; one produced by the platform and signed by a key the build cannot reach is evidence (The Builder Is Inside the Trust Boundary).
- Where it lives matters: a record inside the artefact travels with it and cannot be updated; a record attached in the registry alongside the digest can be queried without pulling the artefact. Most setups want both.
- The verification step is what turns provenance from documentation into a control: a deployment that refuses artefacts without a valid provenance statement from an expected builder (Policy as Code).
The six facts
Each row answers a question someone asks during an incident or a security response. The last column is what happens when that row is missing, which is how you decide which ones to add first.
| Fact | Answers | If missing |
|---|---|---|
| Artefact digest | Which exact bytes are we talking about? | Everything else is attached to a name that can move (Tags Versus Digests) |
| Source commit | What code is this? | You are reading a diff between two versions you cannot identify |
| Builder identity | Who produced it, and can they be trusted? | A hand-pushed artefact is indistinguishable from a CI-produced one |
| Resolved inputs | Which dependency versions are inside? | An advisory means rebuilding everything to find out (Software Bill of Materials) |
| Toolchain | What compiled it, with which flags? | Two builds of one commit differ and you cannot say why (Reproducible Builds) |
| Build timestamp | When, relative to the change that broke things? | Correlating a deploy with a symptom becomes guesswork (Change Correlation) |
The question at 03:14
Provenance is abstract until an incident makes it concrete. The timeline below is the same incident twice: the events are identical, and what the team can do at each step is not.
Notice that provenance does not prevent anything here. It removes the minutes spent establishing what is true, which during an incident is the scarcest resource there is.
- 03:02changeA scheduled release job deploys the artefact tagged
api:2026.8.3 - 03:11signalError rate on the checkout path rises above the alert threshold; on-call is paged
- 03:14actionOperator asks the only useful question: what exactly is running?
- 03:15signalWithout provenance: the tag resolves to a digest nobody recognises. The release job re-ran at 02:40 and moved the tag. Which commit? Unclear
- 03:15actionWith provenance: the running pod reports its digest; the registry attestation names commit
9f2c1a, builderrelease-workflow, and a dependency set - 03:19signalWith provenance:
9f2c1aincludes a transitive dependency bump absent from the previous release — a two-minute query against the recorded input sets - 03:22recoveryRoll back to the previously deployed digest, which is recorded and immutable, rather than to a tag that may have moved
- 03:26recoveryError rate returns to baseline; the investigation continues against a known artefact rather than against a name
Reconstructed to show the shape of the difference, not a report of a specific incident. The step that provenance removes is the one between 03:14 and 03:19, and it is the step that is always longest when nobody wrote anything down at build time.
Where the record lives
The four options are not exclusive and they answer different questions. The one that is almost never sufficient alone is the build log, because log retention is short and log search is slow exactly when you need it.
The build has just finished. Where do the six facts go?
when Always, for the commit and build id — a running process should be able to report what it is.
cost Cannot be updated or revoked, and a compromised build embeds whatever it likes. Also perturbs bit-identity unless the values are declared inputs (Reproducible Builds).
when The default for container images and packages — the record is attached to the digest and queryable without pulling the artefact.
cost Requires a registry that supports attestations, plus signing infrastructure and its key management (Artifact Registries).
when You need to relate several artefacts, a config version and a migration as one release (The Release Manifest).
cost Another system to keep consistent with the registry; drift between the two is worse than either alone.
when Never as the sole record — useful as supporting detail.
cost Log retention is usually far shorter than artefact retention, so artefacts outlive their own history (Artifact Retention).
How to do it properly
Most important first.
- Record all six facts at build time, keyed by the artefact digest, and store them where they cannot be edited afterwards (The Audit Trail).
- Embed the commit SHA and build id in the artefact so a running process can report what it is — a version endpoint that returns the digest is worth more than one that returns a marketing version.
- Attach the record to the artefact in the registry as an attestation, so the digest resolves to its provenance without a database lookup (Artifact Registries).
- Include the resolved dependency set, or a reference to an SBOM that has it (Software Bill of Materials).
- Have the builder sign it, and verify the signature at deploy time rather than only at publish time (Signing and Verifying Artifacts).
- Make the release manifest name digests, not tags, so what a release contains is unambiguous (The Release Manifest).
- Test the query path before you need it: pick a running service and answer all six questions from records alone, without asking a person.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Missing provenance does not cause an outage — it removes your ability to scope one. Containment is that artefacts are immutable and digests are recorded, so the answer exists even when the record does not.
What can go wrong
- Provenance recorded in CI logs only, and log retention is shorter than artefact retention, so old artefacts become anonymous (Artifact Retention).
- A record generated by the build script it describes, which a compromised build can trivially forge.
- Provenance attached to a tag rather than a digest, inheriting every problem tags have.
- Records that exist and are never verified — a deploy path that accepts unsigned, unattested artefacts makes the whole apparatus decorative.
- Manual builds bypassing the path entirely, so the one artefact without provenance is the one pushed during an incident.
- A version endpoint reporting a value baked in from a variable someone forgot to set, which reads as authoritative and is wrong.
- "We have the commit SHA, so we have provenance." The commit says what the source was. It says nothing about which dependencies resolved, which toolchain compiled it, or whether the build was the one your CI ran.
- "Provenance is a compliance requirement." It is an incident-response capability that also satisfies compliance. The value shows up at 3am, not at audit time.
- "The image tag is the version." A tag is a mutable pointer. The digest is the version (Tags Versus Digests).
- "Signing proves the artefact is safe." It proves who produced it. A compromised builder signs malicious artefacts perfectly (The Builder Is Inside the Trust Boundary).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For any running artefact, you can produce the commit, the builder, the toolchain, the dependency set and the build time from records within minutes.
- Deploys of artefacts without valid provenance fail — verified by deliberately trying one in a non-production environment.
- The registry shows an attestation attached to the digest, signed by the expected builder identity.
- The running service reports its own digest, and it matches what the registry says was deployed.
- A dependency advisory can be answered by query — which artefacts contain this version — rather than by rebuilding.
- Provenance is a record, so there is nothing to roll back; the risk is in the *enforcement*. Turning on verification can block deploys of artefacts built before the change, so allow existing digests explicitly rather than disabling verification.
- When enforcement blocks an urgent deploy, the break-glass path must be a recorded exception with an approver, not a flag someone flips (Break-Glass Access).
- If a builder key is compromised, every artefact signed with it becomes unverified. The rollback is re-signing from a clean builder, which only works if the source and inputs are recorded — which is the point.
- Automate generation entirely. Provenance a human types is provenance a human forgets during the incident where it matters.
- Automate verification at the deployment boundary so the record is load-bearing rather than informational.
- Automate the reverse index — dependency to artefacts to environments — because that is the query an advisory actually produces (Scanning, and Why a Finding Is Not a Risk).
- Keep the decision to accept an unverified artefact human, explicit and recorded (Change Management).
- Verification at deploy time adds a dependency on the attestation service to your deploy path. If it is down, you cannot deploy — including a rollback, which needs an explicit answer in advance.
- Storing full input sets per artefact costs storage that grows with build frequency, and retention has to be at least as long as artefact retention.
- Signing introduces key management, which is its own operational burden with its own failure modes (Rotation That Applications Survive).
- Strict enforcement blocks the local build someone needs during an incident — which is exactly the case where the control is most valuable and most resented.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICSLSA defines provenance levels and in-toto defines the statement format; GitHub Actions can produce signed build attestations for artefacts it builds; Sigstore provides keyless signing tied to an OIDC identity. These are specific implementations of one idea — a builder-produced, signed statement bound to a digest — and a registry that only stores tags supports none of them.
- GENERALThe six facts are what any investigation needs regardless of tooling. A text file in an internal registry keyed by digest answers the same questions as a signed attestation, with weaker guarantees against a hostile builder and identical usefulness during an ordinary incident.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.