The question this answers
If staging passed, what exactly did it prove — and is that thing the same thing that reaches production?
A release that has been verified in a lower environment must reach production unchanged, so that the verification means something and so that a rollback is a known-good artifact rather than a rebuild.
One immutable artifact identified by content digest, promoted through environments by changing only which environment references it — so the tested bytes and the running bytes are provably identical.
Rebuilding per environment quietly invalidates the test
The anti-pattern is comfortable and common: each environment's pipeline checks out the release tag and builds. It feels equivalent — same source, same Dockerfile — and it is not, because a build reads more than your source. It resolves a base image tag that moved. It resolves dependency version ranges that published a patch this morning. It runs on a runner with a different toolchain version, a different locale, a different set of cached layers. Three builds from one commit can produce three different sets of bytes, and only one of them was tested.
The failure that follows is memorable precisely because it looks impossible. Staging is green. The production build runs forty minutes later, pulls a base image whose :3.19 tag now points at a new patch release with a changed TLS default, and the service cannot reach an internal endpoint. Nothing in the commit history explains it, because nothing in the commit history changed.
Build once. Produce one artifact, identify it by the content digest rather than by a tag, and promote *that identifier* through environments. The pipeline stops being "build and deploy, three times" and becomes "build once, then three deployments of the same digest".
The digest is the identity; the tag is a label someone can move
A tag is a mutable pointer. v1.4 today and v1.4 tomorrow can reference different content, whether through a re-push after a "tiny fix", a mirror that resolved differently, or a compromised registry account. A digest is the SHA-256 of the image manifest: it either matches or it does not, and it cannot be repointed.
So the deployment manifest should reference the digest. Tags remain useful as human-readable labels and as the thing a person types, but the resolution from tag to digest happens once, at promotion time, and the digest is what is recorded and deployed. This is also what makes signing meaningful — a signature is over a digest, so verifying at deploy time proves the bytes are the ones your build produced.
Two operational consequences fall out. Rollback becomes trivially safe: the previous digest is still in the registry, still signed, still exactly what ran successfully yesterday. And the registry becomes a system of record — "what is running in production?" has an answer that is a hash, and that hash traces to a build, a commit and a reviewer.
# Rebuild-per-environment — three artifacts, one tag, one tested
staging : built 09:14 registry/checkout:v1.4 -> sha256:9c1f2a... ✅ verified
prod : built 09:52 registry/checkout:v1.4 -> sha256:41be77... ❌ never tested
^ base image :3.19 moved at 09:30
# Promote-the-digest — one artifact, three deployments
build : 09:14 registry/checkout@sha256:9c1f2a... signed by build-role
dev : 09:16 deploy sha256:9c1f2a... ✅
staging : 09:31 deploy sha256:9c1f2a... ✅ verified
prod : 10:05 deploy sha256:9c1f2a... ✅ same bytes, verified signature
# Rollback, promote-the-digest:
prod : 10:22 deploy sha256:7e04d1... # yesterday's digest, still in the registry
# no rebuild, no source checkout, no surprisesWhat must be configuration if the artifact is fixed
If one artifact runs in three environments, then everything that differs between environments must come from outside it: endpoints, credentials, feature-flag defaults, log level, replica bounds. This is a constraint that pays for itself — it forces the configuration boundary to be explicit rather than baked in, which is the same argument Configuration Belongs Outside the Image makes from the image side.
The awkward case is anything resolved at build time. A frontend bundle with an API URL compiled in, a binary with an embedded environment name, a config file copied in during the build — each one forces a rebuild per environment and reintroduces the problem. The fixes are mundane: serve configuration at runtime, read it from the environment, or fetch it at startup from a configuration service.
The promotion record itself is worth keeping. Which digest is in which environment, who approved each promotion, when. During an incident the first question is "what changed?", and a promotion log answers it in one line instead of a git archaeology session.
deploy-staging:
script:
- docker build -t registry/checkout:v1.4 . # resolves FROM base:3.19 at 09:14
- docker push registry/checkout:v1.4
- deploy --image registry/checkout:v1.4
deploy-production:
script:
- docker build -t registry/checkout:v1.4 . # resolves FROM base:3.19 at 09:52 — different bytes
- docker push registry/checkout:v1.4 # and it just overwrote the tested one
- deploy --image registry/checkout:v1.4build:
script:
- docker build -t registry/checkout:v1.4 .
- docker push registry/checkout:v1.4
- DIGEST=$(crane digest registry/checkout:v1.4)
- cosign sign registry/checkout@$DIGEST
- echo "$DIGEST" > artifact.digest # the only thing later stages consume
deploy:
parallel:
matrix: [ { ENV: staging }, { ENV: production } ]
script:
- cosign verify registry/checkout@$(cat artifact.digest)
- deploy --image registry/checkout@$(cat artifact.digest) \
--config config/$ENV.yaml # everything that differs lives hereOn the left, production runs bytes nothing tested, and the tested artifact has been overwritten so it cannot even be recovered. On the right, one signed digest moves through the environments and the only thing that varies is a configuration file.
Key points
- Rebuilding per environment produces different bytes from the same commit, so the environment that passed did not test what production runs.
- Identify artifacts by content digest; tags are mutable labels and can be repointed.
- Everything that differs per environment must be configuration supplied at runtime, not baked into the artifact.
- Rollback is trivially safe when the previous digest is still in the registry — no rebuild, no surprises.
- Keep a promotion record: which digest, which environment, who approved. It is the fastest answer to "what changed?".
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A single build produces an artifact; the registry returns its content digest on push.
- • The build identity signs the digest and records provenance linking it to the commit and the build.
- • Each environment's deployment references the digest; promotion is a change to which digest an environment points at.
- • Deploy-time signature verification rejects any artifact not produced by the expected build identity.
- • Rollback re-points an environment at a previously promoted digest, which is still present and still verified.
- • You own registry retention: a rollback target that has been garbage-collected is not a rollback target.
- • You own the promotion record and its approvals.
- • You own the configuration boundary — every value that differs per environment, supplied from outside the artifact.
- • You own build reproducibility hygiene: pinned base image digests and pinned dependency versions, so a rebuild is at least close to deterministic.
- • You own the awkward exceptions, particularly frontend bundles with build-time configuration, which need runtime configuration instead.
- • A moved base-image tag between the staging build and the production build, producing behaviour differences no commit explains.
- • A re-pushed tag overwriting the tested artifact, so the verified bytes no longer exist anywhere.
- • Registry garbage collection removing the previous digest, discovered at the moment you need to roll back.
- • A configuration value baked into the artifact, so promotion silently carries a staging endpoint into production.
- • A promotion that skipped an environment under deadline pressure, which is nearly always the deploy that preceded the incident.
- • Signature verification disabled "temporarily" during an incident and never re-enabled.
- • Registry storage grows with retained digests; retention policy is a direct trade against how far back you can roll.
- • Pull bandwidth at deploy time scales with instance count times image size, which is why Why Image Size Is an Infrastructure Problem is a delivery concern.
- • Promotion coordination is the organisational limit: many services promoting independently is fine; many services that must promote together is a release train, and a much harder problem.
- • A content digest is a supply-chain control: it pins exactly what runs, and a signature over it proves who produced it.
- • Verify signatures at deploy time, not only at build time — the registry is the part of the path an attacker would target.
- • Mutable tags are an attack surface: repointing
v1.4in a compromised registry is invisible to anything that deploys by tag. - • The promotion record is an audit artifact. It answers "who authorised this code to run in production" with a name and a timestamp.
- • Registry storage is usage-shaped and grows with retention; the trade is storage cost against rollback depth.
- • Egress on image pulls is the surprise line item, especially with large images pulled by many instances across zones.
- • Building once instead of three times reduces runner minutes by roughly the number of environments, which is a real if unglamorous saving.
- • Which digest is running in each environment, continuously — and an alert when production runs a digest that never passed staging.
- • Promotion lead time per environment; a long gap between staging and production is where confidence decays.
- • Signature verification failures, which should be zero and should page when they are not.
- • Registry pull latency and failure rate at deploy time, since a slow pull looks exactly like a slow rollout.
- • The signal that lies: the image tag. It tells you what someone called the release, not what is running. Only the digest answers that.
- • For a small service with fast, cheap builds and no staging environment, building at deploy time is not fatal — as long as base images and dependencies are pinned by digest so the build is close to reproducible.
- • Deploying from source (a platform that builds on push) when the platform guarantees the same build is reused across environments — check that it actually does before relying on it.
- • Version-pinned packages instead of images, for non-containerised deployments. The argument is identical: a version that can be re-published is a tag, not a digest.
- • For a single environment, promotion is meaningless and the whole practice is overhead. It becomes valuable exactly when there is something to promote *between*.
- • Buys a meaningful staging signal and a safe rollback; costs a strict configuration boundary and the work of moving build-time values to runtime.
- • Buys immutability; costs registry storage and a retention policy that has to be reasoned about rather than defaulted.
- • Buys a clean audit trail; costs a promotion step that people will want to skip when they are in a hurry.
What people believe, and what is true
Same commit, same build — rebuilding is equivalent.
A build resolves base images, dependency ranges and toolchain versions at build time. The same commit can produce different bytes forty minutes later.
Immutable tags solve it.
They help, and they are still a name that has to be resolved. Deploy by digest and the resolution happens once, at promotion, with a record of the result.
Promotion is just a deploy to the next environment.
It is a deploy of the *same artifact* to the next environment. If a build happens in between, no promotion occurred.