Tags Versus Digests
A tag is a mutable human reference; a digest is content identity. Deploying by digest is what makes a rollout reproducible.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the difference between deploying app:2.3.1 and deploying app@sha256:9f3e..., and when does that difference become an incident?
Every deployment names an artifact. If the name can point somewhere else tomorrow, then the record of what you deployed is a record of what you asked for, not of what you got.
Tags are versions. :2.3.1 is a specific release, :latest is the newest one. Deploying app:2.3.1 deploys version 2.3.1 — that is what the tag means.
A tag is an entry in the registry's naming layer, and in most container registries it can be repointed by anyone who can push. 2.3.1 means whatever it was last pushed as.
- A tag is an entry in the registry's naming layer, and in most container registries it can be repointed by anyone who can push.
2.3.1means whatever it was last pushed as. - Resolution happens at pull time, independently on each node. Two nodes pulling the same tag minutes apart can legitimately receive different bytes, so a fleet can be running two builds under one version number (Version Coexistence: N and N+1, in Both Directions).
- Local caching makes it worse and stranger: a node that already has layers for that tag may not re-resolve at all, so the old bytes persist on some nodes and not others, indefinitely.
- A rollback to a tag reproduces whatever the tag means now, not what it meant when that version was healthy. The recovery action silently becomes a deployment of something new.
- The audit trail records the tag. Six months later, "we deployed 2.3.1" cannot be resolved to bytes, which makes an incident reconstruction unfalsifiable (The Audit Trail).
- The foundations catalogue lists this as the family where every system reports success and the behaviour is from an older build (What Can Fail Between Commit and User). This is the mechanism behind that row.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A digest is a cryptographic hash of the artifact's manifest — the document listing its layers and its config. Change any byte of any layer and the manifest changes, so the digest changes. It is content identity: the name is derived from the thing rather than assigned to it.
- A tag is a reference: a mutable mapping from a human-readable name to a digest, stored in the registry alongside the blobs. Tags exist because
2.3.1is something a person can say andsha256:9f3e...is not. - Pulling by tag is a two-step operation. The client resolves
repo:tagto a digest, then fetches that digest's manifest and layers. Only the second step is verified against content; the first is a lookup whose answer can change. - Pulling by digest skips the mutable step entirely and lets the client verify what it received: the bytes hash to the name it asked for, or the pull fails. That verification is the property that makes a rollout reproducible.
- What a digest does not give you is trust. It guarantees these are the same bytes as before; it says nothing about whether those bytes should be trusted. That is a signature, and it is a separate mechanism (Signing and Verifying Artifacts).
- Multi-architecture images add one level: the tag resolves to an index (a manifest list) whose digest covers the set, and the runtime then selects the per-architecture manifest. Pinning the index digest still pins the whole set.
One tag, two answers
Nothing in this sequence is a bug. Every step behaves exactly as designed, and the result is a fleet running two builds under one version number, with every system reporting success.
The rollback at the end is the part worth sitting with. The operator asks for the last known-good version by name and receives an artifact that has never served traffic.
1Mon 09:00 push app:2.3.1 -> sha256:9f3e... (tested, deployed)2Mon 09:10 10 nodes pull app:2.3.1 -> sha256:9f3e...3 4Tue 16:40 hotfix rebuilt, re-pushed as app:2.3.15 registry name now points -> sha256:c71a...6 running nodes: unchanged, still 9f3e (no reason to re-pull)7 8Wed 11:05 autoscaler adds 4 nodes -> sha256:c71a...9 fleet: 10 x 9f3e + 4 x c71a, all labelled 2.3.110 11Wed 11:30 incident. "roll back to 2.3.0"12 app:2.3.0 -> resolves to whatever it points at todayThe disagreement is invisible from every angle that people normally look at: the deployment spec, the release record, the dashboards and the version endpoint all say 2.3.1. Only the digest distinguishes the two populations.
What each reference form guarantees
Read this as a list of claims you are entitled to make. The most common mistake is treating a digest as a trust statement, and the second most common is treating a version tag as an identity statement.
| Reference | What it identifies | Can change meaning? | Verified by client? | Good for |
|---|---|---|---|---|
app:latest | Whatever was pushed last | Constantly | No | Local development, nothing else |
app:2.3.1 | Whatever that name points at now | Yes, on re-push | No | Humans reading and talking |
app:2.3.1 (immutable tag) | A fixed digest in this registry | Not in this registry; yes in a copy or mirror | No | Reducing surprise, not removing it |
app@sha256:9f3e... | Exactly these bytes | Never | Yes — content is hashed on pull | Deploy manifests, release records, rollbacks |
| Manifest-list digest | A fixed set of per-architecture manifests | Never | Yes | Multi-architecture deploys pinned as one identity |
| Signed digest | These bytes, attested by an identity | Never | Yes, plus signature verification | Deciding whether to trust, not just what to run |
Where the digest has to appear
Resolving a tag once and then carrying the digest is a small pipeline change with an outsized effect: it converts "which version is running" from a question requiring judgement into a string comparison.
The four places below are the ones that must agree. When they do, an incident starts with diagnosis. When any two disagree, you have found the problem before you started looking for it.
- 1Build output
The push returns the digest the registry stored.
fails by The pipeline captures the tag it pushed instead of the digest the registry returned.
evidence The digest is a pipeline output variable, not a log line.
- 2Release record
Names the digest, the config version and any migration in the release.
fails by Records a tag, making the record unresolvable later (The Release Manifest).
evidence The record contains an
@sha256:reference. - 3Deploy manifest
The thing actually applied to the platform references the digest.
fails by A templating layer substitutes a tag because that is what the template took.
evidence The applied spec, read back from the platform, contains the digest.
- 4Running workload
Reports the digest it is running, alongside its commit.
fails by No version endpoint, so the running value must be inferred.
evidence The value reported by the process matches the release record (A Successful Deploy Is Not Evidence of a Healthy System).
The comparison is the deliverable. Three systems that each faithfully report a different thing are worth less than three systems reporting one string you can diff.
How to do it properly
Most important first.
- Resolve the tag to a digest once, in the pipeline, and carry the digest through every subsequent stage. Human-facing systems can show the tag; machines should pass the digest.
- Record the digest in the release record, the deploy annotation and the runtime version endpoint, so all three can be compared during an incident (The Release Manifest).
- Pin base images by digest too. A
FROMon a floating tag makes your build non-reproducible for reasons that have nothing to do with your code (Reproducible Builds). - Enable immutable tags where the registry supports them. It does not remove the need to deploy by digest; it removes one way of being surprised.
- Keep tags for humans: a semantic or build-number tag for reading, plus a moving
:latestif your workflow wants one. Just never let a deploy path consume them. - On a platform that caches images per node, be explicit about the pull policy so a tag deploy cannot silently reuse local layers — and prefer digests, which make the question irrelevant.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A rolling deployment limits how fast the wrong bytes spread, and nothing limits how long they persist: nodes that already hold the tag's old layers can keep serving them until something forces a re-pull (Rolling: Two Versions, One Database).
What can go wrong
- Deploying by tag and discovering during an incident that the version numbers agree and the behaviour does not.
- Rebuilding and re-pushing the same tag to "fix" an artifact, which leaves part of the fleet on the old bytes until something restarts.
- A digest recorded in the release record but a tag used in the actual deploy manifest, so the record is accurate about a decision nobody executed.
- Pinning a digest and then never updating it, so a base image with a known vulnerability is pinned in place for a year (Dependency Security in security terms).
- The mitigation failing in a subtle way: a pull-through cache keyed on tags in front of a digest-based deploy, which reintroduces mutable resolution one layer down.
- Assuming a digest match proves provenance. It proves byte equality with something you saw before, which is a different and weaker claim than "this came from our build system".
- "Immutable tags solve it, so digests are unnecessary." Immutable tags stop the tag being repointed in that registry. They do not help with a mirror, a cache, or a copy into a different registry, and they do not give the client anything to verify against.
- "
:latestis the only dangerous tag." Any mutable tag is.:latestis just the one that is obviously mutable, which makes it less dangerous than a version tag people trust. - "The digest proves the artifact is safe." It proves the bytes are the bytes. Trust requires a signature over that digest by an identity you accept (Signing and Verifying Artifacts).
- "We use digests, so our builds are reproducible." Deploying by digest makes the *rollout* reproducible. Making the *build* reproducible is a separate property about inputs (Reproducible Builds).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The digest in the release record, the digest in the deploy manifest and the digest reported by the running workload are the same string.
- Deploying the same release twice produces no change at all, because the identity is already what is running.
- A pull of a tampered artifact fails at the client rather than succeeding quietly, because the content did not hash to the requested digest.
- Every image reference in the deployment — application, sidecars, init containers, migration jobs — contains an
@sha256:and not only a tag.
- Rollback by digest is exact: the previous digest is a set of bytes that demonstrably served traffic. Rollback by tag is a request that may be answered differently than it was last time.
- This is the single highest-leverage change in the module for recovery time. It costs a pipeline variable and it converts "roll back to the last good version" from a hope into an operation.
- The one thing digests cannot roll back is a deletion. If the older manifest was garbage-collected, the digest is a name for something that no longer exists (Artifact Retention).
- Automate tag-to-digest resolution at the earliest possible point and make everything downstream take a digest as input.
- Automate a pre-deploy assertion that the digest about to be deployed is one that an earlier environment recorded (Build Once, Deploy Many).
- Keep base-image digest bumps automated but reviewed: a bot proposing the new digest is good, a pipeline silently following a floating tag is the thing you were avoiding.
- Digests are unreadable. Any human-facing surface — dashboards, release notes, chat notifications — needs the tag alongside, which is extra plumbing.
- Pinning by digest means base image updates become explicit work rather than something that happens on its own. That is the point, and it is still work.
- Some tooling and some registry UIs are tag-oriented and make digest workflows awkward. That friction is real and is not a reason to deploy by tag.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICTag mutability is a registry policy. Most OCI registries allow re-pushing a tag by default and some offer an immutability setting per repository; most language package registries forbid republishing a version outright, which is why the tag-versus-digest distinction feels foreign to people coming from npm or PyPI.
- PLATFORM-SPECIFICWhether a tag deploy re-resolves at all depends on the runtime's image pull policy and its local layer cache. A platform that pulls only when the image is absent will keep serving old bytes for a moved tag indefinitely; one that always pulls will re-resolve on every start. Deploying by digest makes both behave identically, which is the real argument for it.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.