Signing and Verifying Artifacts
A trusted builder signs the artifact it produced, and the deployment refuses anything whose signature it cannot verify — the verification is the control, not the signature.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does a deployment know that the artifact it is about to run came from our pipeline and not from somewhere else?
A registry accepts pushes from whatever has credentials. A deployment pulls whatever the reference resolves to. Neither step, by itself, establishes where the artifact came from.
Restrict who can push to the registry and treat everything in it as trusted. If it is in our registry, we built it.
Registry credentials are held by every build job, and often by more identities than anyone has enumerated. "We pushed it" becomes "something with push access pushed it".
- Registry credentials are held by every build job, and often by more identities than anyone has enumerated. "We pushed it" becomes "something with push access pushed it".
- Tags are mutable in most registries by default. An artifact that passed every gate under a tag can be replaced afterwards without any deployment noticing (Tags Versus Digests).
- A registry-level trust boundary provides no evidence after the fact. During an incident you cannot answer "which build produced this" from the artifact alone.
- It gives you nothing across boundaries: an artifact received from a vendor, a partner, or another business unit arrives with no attestation at all.
- Restore a registry from a backup, mirror it to another region, or migrate it, and the "we control the registry" assumption quietly does not follow the artifacts.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A signature binds an identity to a specific artifact digest. Signing a tag signs a pointer, which is worth very little; signing a digest binds the identity to exact content.
- Verification recomputes the digest of what was received, checks the signature against a public key or certificate, and checks the signer identity against a policy — "this repository, built by this workflow" rather than merely "some valid signature".
- Long-lived key signing means a key exists somewhere and must be protected, rotated and revoked. Keyless signing instead issues a short-lived certificate bound to the builder's workload identity and records the signature in a public transparency log, which moves the problem from key custody to identity and log integrity (Workload Identity).
- The control is the verification at admission. Signing produces evidence; refusing to run unverified artifacts is what makes the evidence load-bearing.
- A signature says who produced the artifact. It says nothing about whether the contents are safe. Both statements are needed and they are answered by different controls (Scanning, and Why a Finding Is Not a Risk, Software Bill of Materials).
The chain that makes a signature mean something
Each step has one job and one way of being useless. The pattern to look for is a step that produces evidence with no consumer.
- 1Build on a trusted builder
Produce the artifact in an isolated environment whose definition came from reviewed source.
fails by A builder that build steps can influence, so the thing being signed is not what was reviewed.
evidence The build definition is in the protected branch and the runner is ephemeral (The Builder Is Inside the Trust Boundary).
- 2Compute the digest
Derive a content address for the exact bytes produced.
fails by Nothing much — this step is reliable and is what everything else hangs from.
evidence The digest appears in the build log and in the release record.
- 3Sign the digest
Bind the builder identity to that content, using a key or a short-lived identity-bound certificate.
fails by Signing a tag instead, or signing with an identity the build steps can also use.
evidence The signing identity is not present in the build environment's credentials.
- 4Publish artifact and signature
Store both, addressed by digest.
fails by Signature stored somewhere the deploy path cannot reach.
evidence Verification works from the deployment environment, not just from CI.
- 5Promote by digest
Move the same digest through environments rather than rebuilding (Build Once, Deploy Many).
fails by Rebuilding per environment, which produces a different artifact from the one that was tested and signed.
evidence The digest in production matches the digest that passed staging (Promotion).
- 6Verify at admission
Check signature validity and signer identity against policy before anything runs.
fails by Not enforced, enforced only in staging, or accepting any valid signature.
evidence An unsigned artifact has actually been rejected in the environment that matters.
What a verification policy has to say
The difference between a policy that helps and one that is decorative is whether it constrains who signed, not merely whether something signed. Written out as a rule, the distinction is obvious; expressed as a checkbox in a tool, it is easy to miss.
1decorative:2 accept if: signature verifies3 -- any identity with any signing capability passes.4 includes every other pipeline in the org, and5 anything that obtained a signing identity.6 7useful:8 accept if: signature verifies9 AND signer identity == our release workflow10 AND source repository == our repository11 AND artifact referenced by digest, not tag12 otherwise: refuse to run, and emit an event13 14 emergency: a named, time-boxed exception, logged,15 expiring automatically -- never a16 permanent bypass flagThe last block is the one that decides whether the control survives its first incident. An exception path that has to be invented at 03:00 gets invented as a permanent flag.
Where signing programmes actually fail
Almost none of these are cryptographic. They are gaps between the part that produces evidence and the part that is supposed to act on it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Signing rolled out; verification "next quarter" | Dashboards show 100% signed artifacts | The signing half is easy and visible; the verification half blocks deploys | Enforce in one low-risk environment immediately, then expand. Unverified signing is not a partial control, it is no control |
| Deploy references a tag | Verification passes, contents differ from what was tested | The tag was repointed after the gate | Deploy by digest end to end; treat any tag in a deployment manifest as a defect |
| Verifying service unavailable | All deploys blocked during an unrelated incident | Fail-closed with no considered fallback | Decide the failure mode in advance per environment, and monitor the verifier as a production dependency |
| Emergency bypass used | Nothing; it works and nobody revisits it | A bypass with no expiry | Time-box it at the point of use and alert while it is active (Deploys on the Same Timeline as the Symptom) |
| Build step can read the signing identity | No symptom until something goes wrong | Signing performed inside the build rather than by the builder | Move signing out of reach of build steps; this is the core property of a trusted builder |
| Third-party artifact deployed | Refused, or silently exempted | The policy assumed everything is built in house | Decide explicitly: mirror and re-sign after review, or add a narrowly scoped exemption with an owner |
How to do it properly
Most important first.
- Sign at the builder, with an identity the build steps themselves cannot use. A signing capability reachable from inside the build is a capability an influenced build has (The Builder Is Inside the Trust Boundary).
- Sign the digest, deploy by digest, and verify the digest at admission. Any place a mutable tag survives in that chain is where the control leaks.
- Verify identity, not just validity. The policy should name the expected signer — repository, workflow, environment — so an artifact signed by a different pipeline in the same organisation is refused.
- Enforce at the deployment boundary, on the path everything takes. Verification implemented only in the standard pipeline is bypassed by every non-standard deploy (Guardrails, Not Gates).
- Design the emergency path before you need it: a documented, logged, time-boxed way to deploy an unverified artifact, rather than an undocumented one improvised during an incident (Break-Glass Access).
- Verify the verification. Push a deliberately unsigned artifact to a non-production environment and confirm it is refused.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Admission verification is itself the containment. Where it is not enforced, nothing downstream distinguishes a pipeline artifact from any other.
What can go wrong
- Signing enabled, verification never enforced. Extremely common, produces dashboards, provides no security.
- Verification enforced in one environment and not in the one that matters, because enabling it in production would have blocked a release.
- A policy that accepts any valid signature, so any identity with a signing capability anywhere is accepted.
- Break-glass bypass left enabled after the incident, which silently disables the control for everything afterwards.
- Verification failing closed with no clear error, so operators disable it under time pressure rather than debug it.
- Signing key or signing identity available to build steps, so an influenced build signs its own output (Securing the Pipeline Itself).
- Signatures stored somewhere the runtime cannot reach during a network incident, turning a security control into an availability dependency.
- "Signed means safe." Signed means attributable. A vulnerable or malicious artifact from a compromised builder is signed correctly.
- "We sign our images, so we are covered." Signing without enforced verification changes nothing an attacker has to work around.
- "The signature protects the artifact in the registry." It lets a consumer detect substitution. It does not prevent a push, and it does not prevent a tag being repointed — deploying by digest is what removes that.
- "Verification means we do not need to scan." They answer different questions: provenance versus contents. A correctly signed artifact can contain a critical vulnerability.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- An unsigned artifact was rejected by production admission control, observed in a test rather than assumed from configuration.
- Every running workload's digest has a signature that verifies against the expected builder identity, checkable at any time.
- The count of admission bypasses is visible, and each one has an owner and a reason (The Audit Trail).
- Rolling back to a previous artifact is unaffected: it was signed when it was built, so verification succeeds without special handling.
- Disabling verification to unblock a deploy is a change to the trust boundary. Treat it as a production change with an approval and an expiry, not as a pipeline setting.
- Automate: signing at the builder, digest-based promotion, verification at admission, and alerting on any bypass.
- Keep human: approving a bypass, and any change to which signer identities the policy accepts. The second one is the whole boundary expressed as a list (Change Management).
- Admission verification adds a dependency in the deploy path. If the verifying service or the transparency log is unavailable, you must decide in advance whether to fail open or fail closed — and both answers are defensible for different environments (Fail Open vs Fail Closed in Security Engineering sets out the trade).
- Keyless signing removes key custody and adds a dependency on an identity provider and a public log, including the fact that the log is public.
- Signing everything produces a lot of metadata to store and expire alongside artifacts (Artifact Retention).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICDescribed for container images, where digests, detached signatures and admission controllers all exist. Language package ecosystems differ substantially — some support publisher attestations, some support nothing beyond a checksum — and a signing strategy that works for images does not transfer automatically to your packages.
- PLATFORM-SPECIFICWhere verification is enforced depends on the platform: an admission webhook in an orchestrator, a policy on the deployment service, or a check in the pipeline as a weaker last resort. The pipeline-only version is bypassed by anything that deploys another way, which is the gap worth checking for.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — transparency logs as append-only, independently auditable records, and what "the log is public" means for what you put in it.