ArtifactsTOOL-SPECIFICGENERAL
Artifact Registries
The store artifacts live in between build and deploy — and a piece of production infrastructure on the critical path of every scale-up.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
Where do artifacts live between being built and being run, and what does that store owe you when it is 3am?
The problem
A build output on the CI runner is one disk failure and one runner recycle away from gone, has no access control, no immutability guarantee, and no way for a hundred machines to fetch it at once.
What teams do first
Publish build outputs to a bucket, or keep them on the build server and copy them to hosts at deploy time. It is just file storage — a registry is an extra system to run.
How it breaks
Nothing enforces immutability. A path in a bucket can be overwritten, and the overwrite is invisible to anything holding the path.
How it breaks in production
- Nothing enforces immutability. A path in a bucket can be overwritten, and the overwrite is invisible to anything holding the path.
- There is no content addressing, so "the artifact we tested" is a filename and a promise (Tags Versus Digests).
- No deduplication. Every build of a container-shaped artifact re-uploads the whole filesystem, including the base layers that have not changed in months.
- No authorisation model that distinguishes push from pull, so anything that can read can also replace.
- Retrieval is not designed for fan-out. A rollout that starts a hundred instances at once produces a hundred simultaneous fetches, and a plain object store with no local caching turns that into a bandwidth event during a deploy (Egress: Moving Data Costs Money, Not Just Storing It in cloud terms).
- The moment you depend on it for scale-up, it is production infrastructure with no availability target and nobody on call for it.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A registry is three things stacked: a content-addressed blob store holding the bytes, a naming layer mapping human references to content identities, and a metadata and policy layer handling authentication, immutability rules, retention and replication.
- For container images specifically, the blob store holds layers and manifests. A manifest lists the layers and the config; the manifest's hash is the image digest. Layers are shared: two images built from the same base share those blobs, so a push uploads only what is new (Layers and the Build Cache).
- The naming layer is where mutability lives.
repository:tagis an entry pointing at a digest, and in most registries that entry can be repointed. The blobs are immutable; the names are not, and every "wrong version deployed" incident lives in that gap. - The pull path is on the critical path of more than deploys. A node replacement, an autoscaling event, an eviction and a crash-loop restart on a fresh node all pull. A registry outage does not roll anything back — it prevents you from adding capacity, which is a far worse failure during an incident.
- Language package registries (npm, PyPI, Maven Central, crates.io) are the same three layers with different policies: most forbid republishing a version at all, which gives immutability at the naming layer that container registries generally do not.
What is actually stored, and what is only a name
TOOL-SPECIFICThe split between mutable names and immutable blobs is an OCI registry property. Language package registries fold the two together by refusing to republish a version; generic file stores have neither, which is why a bucket is not a registry.
The single most useful thing to hold in your head is which parts of a registry are immutable and which are not. The blobs are content-addressed and cannot change meaning. The names can be repointed at any time by anyone who can push.
This is not a flaw. Mutable names are how :latest, :stable and :v2 stay useful. It only becomes a failure when something deploys by name and assumes it got the bytes it tested (Tags Versus Digests).
UserLLMAgentToolDataDecisionHumanGuardrail
Store types, and what each one actually gives you
These are not interchangeable, and the differences are precisely the properties you rely on during an incident. The last column is the one to check before choosing.
| Store | Addressing | Immutability | What differs operationally |
|---|---|---|---|
| OCI container registry | Digest, plus mutable tags | Blobs immutable; tags usually mutable unless configured | Layer sharing makes pushes cheap and garbage collection subtle |
| Language package registry | Name + version | Version republish usually forbidden outright | Yanking hides a version without deleting it; caches keep serving it |
| Generic artifact repository | Path, sometimes checksum | Whatever policy you configure | Flexible enough to lose the guarantees you wanted |
| Object storage bucket | Path | None by default; versioning is opt-in | No pull auth model tied to workload identity; no dedup |
| Machine image catalogue | Provider-assigned image id | Immutable once registered | Region-scoped: an image must be copied to each region, and the copy has a new id |
The registry failures that show up as something else
Almost every registry failure is first reported as a deploy failure or a capacity failure, because that is where it becomes visible. Recognising the signature saves the ten minutes usually spent debugging the application.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Registry unreachable | Existing traffic fine; new instances stuck starting; autoscaler adds nothing | Pull path down or unroutable from the workload network | Confirm from a node, not from a laptop; fall back to a cache or replica; do not roll back the app, it is not the app |
| Pull rate limit | Intermittent pull failures that grow with rollout size | Shared or anonymous quota on a public registry | Authenticate pulls, mirror the base images you depend on |
| Expired pull credential | Workloads start fine for weeks, then a rescheduled one cannot pull | Short-lived token captured at creation time and never refreshed | Use workload identity rather than a stored token (Workload Identity) |
| Garbage collection ran | A deployment that was healthy cannot be rescheduled; rollback target missing | Untagged manifest referenced only by digest was collected | Exclude anything deployed or rollback-eligible from GC (Artifact Retention) |
| Tag repointed | Deploy reports success; behaviour is from a different build | Deploying by tag while something re-pushed the tag | Deploy by digest; enable immutable tags (Tags Versus Digests) |
| Cross-region pull | Cold starts slow everywhere except the registry's own region | Single-region registry serving multi-region workloads | Replicate the registry, or pre-pull to nodes as part of provisioning |
| Push succeeded, metadata did not | Artifact present with no provenance or scan result | Push and attestation are separate steps and only one is required | Make the publish step atomic: no artifact without its metadata (Build Provenance) |
How to do it properly
Most important first.
- Treat the registry as production infrastructure: it needs an availability expectation, monitoring, and a plan for what happens when it is unreachable.
- Turn on immutable tags if the registry supports them, and deploy by digest regardless (Tags Versus Digests).
- Separate push and pull credentials. Build systems push; runtime identities pull and nothing more (Workload Identity).
- Cache or replicate close to where things run. A pull-through cache in the same network as the workloads removes the registry from the scale-up path for anything already pulled.
- Scan and sign at push time so the metadata travels with the artifact rather than living in a separate system (Signing and Verifying Artifacts, Scanning, and Why a Finding Is Not a Risk).
- Know what your registry's garbage collection deletes and when, before you find out during a rollback (Artifact Retention).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Instances already running are unaffected — the registry is not on the request path. What is contained is only the current fleet; anything requiring a new instance is blocked, so the impact grows with time and with any failure that kills capacity.
What can go wrong
Failure modes, including of the mitigation
- Registry unavailable during an incident: existing instances keep serving, new ones cannot start, and the autoscaler quietly fails to add capacity.
- Rate limiting on a shared or public registry, which appears as intermittent pull failures during large rollouts and looks like a network problem.
- Pull credentials that expire — often a token with a short lifetime that was working when the deployment was created and is not when a pod is rescheduled at 4am (Rotation That Applications Survive).
- Garbage collection removing an untagged manifest that a running deployment still references by digest. Nothing breaks until a node is replaced, and then the workload cannot be rescheduled.
- A registry in one region serving workloads in another, adding cross-region transfer to every cold start and a hard dependency on that region.
- The mitigation failing: a pull-through cache that caches by tag rather than by digest, which reintroduces the mutable-name problem it was supposed to hide.
Misreads this invites
- "The registry is just storage." It is on the path of every scale-up and every node replacement. Storage that production cannot start without is production infrastructure.
- "Layers are deduplicated so storage is basically free." Deduplication is per-repository in some registries and per-registry in others, and the metadata for a very large number of manifests has its own cost.
- "Deleting a tag deletes the image." In most container registries it removes a name. The manifest may survive untagged, and the blobs survive until garbage collection — which is both why deletion does not free space immediately and why an untagged manifest can still be pulled by digest.
- "Our registry has never been down." Neither had anyone else's, until the deploy that needed it.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A fresh node with no local layers can pull and start the current artifact — tested deliberately, not discovered during an outage.
- Registry availability and pull error rate appear on the same dashboard as the deploy pipeline, because a pull failure presents as a deploy failure.
- The digest read back from the registry matches the digest the build pushed.
- Pull identities cannot push, verified by trying.
How you get back
- Rolling back a bad artifact means pulling an older one, which requires that the older one still exists and that its layers were not garbage-collected. Retention policy is therefore part of your rollback plan, not a cost-control detail.
- You cannot roll back a registry outage. The mitigations are local caches, replicas and, for the truly critical path, nodes that already hold the layers they need.
What to automate, and what stays human
- Automate push, tagging, scanning, signing and metadata attachment as one step, so an artifact never exists in the store without its provenance.
- Automate retention as policy over metadata rather than as a periodic manual cleanup, and make the policy refuse to delete anything currently deployed.
- Do not automate deletion of anything a running or recently-running release references. That deletion should require a human who can be asked "are you sure nothing is on this".
What this costs
- A managed registry is one less system to operate and one more external dependency on your scale-up path, usually in a specific region.
- Self-hosting gives you control over retention, replication and availability, and gives you a storage system to operate that is now load-bearing for every deploy.
- Pull-through caches reduce registry load and add a layer that can serve stale content if it is keyed on mutable names.
- Immutable tags remove a whole class of incident and remove the convenience of moving
:latest, which some workflows genuinely rely on.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICRegistries differ in exactly the places that cause incidents. Some support immutable tags and some do not; some garbage-collect untagged manifests automatically on a schedule and some only on an explicit run; some deduplicate blobs registry-wide and some per repository; managed cloud registries apply their own retention rules and their own per-account rate limits. Read your registry's garbage collection and tag-mutability documentation specifically — the general model in this lesson tells you what to ask, not what your registry answers.
- GENERALThe three-layer structure — blobs, names, policy — holds for OCI registries, language package registries and generic artifact stores alike. Language registries typically forbid republishing a version, which is the same immutability guarantee enforced at a different layer.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.