The question this answers
How does a commit become the exact artifact that production runs, and where does that chain usually break?
A merged pull request must reach production automatically, and the team must be able to prove — during an incident, at two in the morning — that the image serving traffic is the one that passed the tests.
A single, auditable path from commit to running workload, where every stage names its input, its output, the identity that performed it, and what it would look like if it went wrong.
Six states, and what breaks at each one
The pipeline is a state machine, and treating it as one is what makes it debuggable. Each state consumes a specific artifact and produces a specific artifact. When a deployment goes wrong, the useful first question is never "why is production broken" but "which state produced something different from what the next state expected".
The rule that ties the states together is build once, promote the same bytes. A rebuild for staging and a second rebuild for production are two different artifacts however identical the inputs looked, because an unpinned base or a floating dependency can differ between them. Once staging tests one artifact and production runs another, the tests have stopped being evidence. See Build Once, Promote the Same Bytes.
- 1Source commitinstant
A merged commit on the main branch, with a lockfile and a Dockerfile alongside the code.
The Dockerfile lives in a different repository than the code it builds, so the two drift and nobody notices until a build fails.
- 2CI triggereds
A runner checks out the commit, assumes a build identity, restores the layer cache.
The runner has broad standing credentials and runs untrusted pull-request code — the classic supply-chain foothold.
- 3Build1–10 min
The builder executes the instructions and produces layers plus a config document.
Cache misses from bad instruction order; unpinned base or packages making the output a function of the calendar.
- 4Image + digests
A manifest whose digest is the artifact's immutable identity. Scanned, and labelled with the source commit.
Only a mutable tag is recorded, so the artifact can be replaced later and nothing detects it.
- 5Registry push10 s – 2 min
Missing layers are uploaded; the registry returns the digest. This is the promotion boundary.
Push permissions granted to anything that can run a job, so any workflow can publish an image production will pull.
- 6Deployment30 s – 5 min
The platform pulls that digest onto hosts, starts containers, waits for health checks, shifts traffic.
The deployment references a tag rather than the digest, so what runs is whatever the tag points at right now.
Two pipelines, one of which cannot prove anything
The difference below is the whole lesson compressed into twenty lines. On the left, each environment rebuilds from source and deploys a mutable tag: the artifact tested is not the artifact shipped, and "roll back to the previous version" has no defined meaning because the tag has already moved.
On the right, one build produces one digest. Testing, staging and production all reference that digest, and promotion is a metadata change rather than a rebuild. Rollback becomes "deploy the previous digest", which is instant and exact.
staging:
script:
- docker build -t app:latest . # build #1
- docker push app:latest
- kubectl set image deploy/app app=app:latest
production:
when: manual
script:
- docker build -t app:latest . # build #2 — different bytes, same name
- docker push app:latest # overwrites what staging tested
- kubectl set image deploy/app app=app:latest
# rollback: undefined. :latest no longer points at the previous release.build:
script:
- docker build -t app:${CI_COMMIT_SHA} .
- docker push app:${CI_COMMIT_SHA}
- DIGEST=$(crane digest app:${CI_COMMIT_SHA}) # the real identity
- echo "$DIGEST" > artifact.digest
artifacts: { paths: [artifact.digest] }
staging:
script: [ "kubectl set image deploy/app app=app@$(cat artifact.digest)" ]
production:
when: manual # a gate, not a rebuild
script: [ "kubectl set image deploy/app app=app@$(cat artifact.digest)" ]
# rollback: deploy the previous digest. Exact, instant, auditable.The left pipeline can never answer "is production running what we tested?". The right one answers it with a string comparison. Everything else — scanning, signing, approval gates — only means something once the artifact has a stable identity.
The pipeline is an identity with production access
The build pipeline is frequently the most privileged component in an organization, and the least examined. It can read all source, it holds registry credentials, and it can change what production runs. An attacker who compromises it does not need to find a vulnerability in your application — they can simply ship one.
The structural fix is separation. The job that builds untrusted code — a fork's pull request — gets no secrets and no push permission. The job that publishes gets push-only credentials, scoped to one repository, and cannot delete or overwrite. The deployment identity gets pull-only, and never gets build permissions. This is the same least-privilege argument as Least Privilege in Infrastructure, applied to the one identity most teams leave broad because narrowing it is inconvenient.
- registry:PushImage on checkout/* (new tags only)
- registry:PullImage on checkout/*
- registry:GetAuthorizationToken
- Push a new tag to one repository, and read back the digest it produced.
- registry:DeleteImage — nothing in a build should remove an artifact
- registry:PushImage on any other repository
- deploy:UpdateWorkload — building and deploying are different identities
- secretsmanager:GetSecretValue on production/* — the builder never needs production credentials
Blast radius: A compromised runner can publish a new image tag in one repository. It cannot overwrite an existing digest, cannot deploy, and cannot read production secrets — so the malicious artifact still has to pass the approval gate and be pulled deliberately.
Key points
- Source → Dockerfile → build → image → registry → deployment: six states, each with a distinct input, output, identity and failure mode.
- Build once and promote the same digest. A per-environment rebuild silently invalidates every test that ran before it.
- The digest, not the tag, is the artifact's identity. Deploy the digest and rollback becomes exact.
- The pipeline is a privileged identity: it reads all source, holds registry credentials and decides what production runs.
- Separate the identity that builds untrusted code, the identity that pushes, and the identity that deploys. Nothing should hold all three.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A merge event triggers a runner, which checks out the commit and assumes a short-lived build identity rather than using a static key.
- • The builder restores a layer cache, executes the instructions, and produces layers plus a config document; unchanged layers are reused.
- • The image is labelled with the source commit and scanned; the resulting manifest digest becomes the artifact's permanent name.
- • Push uploads only the layers the registry lacks and returns the digest, which the pipeline records as an output artifact.
- • Deployment references that digest; hosts pull it, start containers, and the platform waits for readiness before shifting traffic.
- • Promotion to the next environment re-uses the recorded digest — a metadata change, with no build step involved.
- • You own build reproducibility: pinned base by digest, a committed lockfile, and pinned system packages, or the pipeline is not deterministic.
- • You own the cache, including its invalidation. A stale cache that keeps serving an old dependency layer is a genuinely nasty class of bug.
- • You own the runner's identity, its lifetime and whether it is ephemeral. A persistent runner accumulates state and credentials between jobs.
- • You own the record linking digest → commit → test run. Without it an incident cannot establish what was actually deployed.
- • You own build capacity, which becomes the delivery bottleneck long before runtime does.
- • A dependency registry is briefly unavailable and the build fails; every deploy in the organization is blocked on a third party nobody listed as a dependency.
- • The layer cache holds a dependency layer built before a security patch; builds keep succeeding and keep shipping the vulnerable library.
- • Staging passes and production fails because each environment rebuilt from source and the base image moved between the two builds.
- • The deployment pulls a tag that has been overwritten, so a rollback deploys the same broken bytes under a different name.
- • A build succeeds and pushes, but the deployment fails to pull because the node's identity has no permission on that repository — the image-pull error that looks like a network problem.
- • Build time scales with cache hit rate, not with team size — until concurrency contention on shared runners becomes the real queue.
- • A monorepo scales badly without path filters: every commit rebuilds every service, and pipeline duration grows with the organization.
- • Registry write throughput and rate limits become visible when dozens of pipelines push simultaneously, usually right after a merge freeze ends.
- • What runs out first is almost always human: a pipeline slow enough to context-switch away from is a pipeline people stop watching.
- • Untrusted code — a fork's pull request — must run in a job with no secrets and no push permission. This is the most common CI compromise path.
- • Use short-lived, workload-identity-based credentials rather than long-lived registry keys stored as CI variables — see Roles vs Static Keys.
- • Record provenance: which commit, which builder, which inputs. Signing the artifact makes that record verifiable at deploy time.
- • Never pass secrets as build arguments. They are recorded in the image config and readable by anyone who can pull it — see Configuration Belongs Outside the Image.
- • Build minutes are the direct meter, and cache hit rate is the dominant lever on it.
- • Registry storage grows with every retained build; a retention policy is a cost control that also reduces the number of images an attacker could target.
- • Egress for pulls, especially across regions or from a public registry through a NAT path — the same meter NAT Gateway and Egress: Moving Data Costs Money, Not Just Storing It describe.
- • The largest cost is usually not on any invoice: engineers waiting on a slow pipeline several times a day.
- • Pipeline duration split by stage, so "the build is slow" resolves to a specific instruction or a specific queue.
- • Cache hit rate per build, the leading indicator of both duration and cost.
- • Deployed digest versus expected digest, checked continuously — the single most useful drift signal in delivery.
- • The signal that lies: a green pipeline. It proves the steps exited zero, not that the artifact is the one under test or that the deployment took effect.
- • Build locally and push manually for a one-person project. It is honest, it works, and a pipeline that nobody maintains is worse than no pipeline.
- • A platform that builds from a git push — a PaaS or a buildpack-based service — when the team does not want to own build infrastructure at all.
- • A managed build service instead of self-hosted runners, which removes runner patching and persistent-state risk at the cost of some control.
- • For an infrequently-changing service, a scheduled weekly build is often enough. Per-commit builds are a response to change frequency, not a rule.
- • Automation buys repeatability and costs a system that must itself be maintained, secured and debugged — usually during an incident.
- • Build-once-promote buys provable artifacts and costs a slightly more complex pipeline and somewhere to record digests.
- • Aggressive caching buys speed and costs correctness at the margins; a cache that is never invalidated ships stale dependencies indefinitely.
- • Separating build, push and deploy identities buys a much smaller blast radius and costs the convenience of one token that does everything.
Follow a build: source to running container
app:latest a name someone can repoint at any time
sha256:67467ed57c35… the bytes themselves — nobody can repoint content
tag → digest is a lookup, and the lookup happens at pull time,
once per node, which is why "identical" nodes can differWhat people believe, and what is true
Rebuilding from the same commit gives the same image.
Only with a digest-pinned base and fully pinned installs. Otherwise the build is a function of when it ran.
The pipeline is infrastructure plumbing, not a security concern.
It reads all source, holds registry credentials and can change production. It is usually the highest-value target in the organization.
A green pipeline means the deployment succeeded.
It means the steps exited zero. Whether the new digest is actually serving traffic is a separate observation.