From Developer to Users
The full path a change travels, as a model you can debug against when something in it goes wrong.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the complete chain between a developer and a user, and why is knowing it a debugging tool?
When production breaks, the useful first question is where in the chain it broke — but only if you have the chain in your head.
The chain is: push code, it deploys, users get it. The middle is the platform's job.
Each hop in the middle is a separate system with its own failure modes, and roughly half of production incidents originate there rather than in application code.
- Each hop in the middle is a separate system with its own failure modes, and roughly half of production incidents originate there rather than in application code.
- Without the model, every delivery failure looks the same: "the deploy did not work". With it, you can localise in seconds.
- The failures are often invisible from the application side — a stale artifact, a config that did not update, a registry permission — and produce symptoms that look like code bugs.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The chain is: developer → git → CI → build → artifact → registry → deployment system → infrastructure → application → users.
- Each hop transforms or moves something, and each can fail independently: the build can succeed with a wrong dependency, the registry can serve an old digest, the deployment system can apply successfully while the pods never become ready.
- Debugging production delivery is walking this chain and asking, at each hop, "did the thing I expect actually arrive here?"
The chain, and what fails at each hop
This is the model to debug against. Each row is a hop, what it produces, and the failure that is characteristic of it.
| Hop | Produces | Characteristic failure |
|---|---|---|
| Developer | A commit | Change larger than can be reviewed meaningfully |
| Git | A ref that may become production | Merged to the wrong branch; protected-branch rule missing |
| CI | A verdict | Green because the relevant check does not exist |
| Build | Compiled output | Unpinned dependency resolves differently than yesterday |
| Artifact | An immutable package | Rebuilt per environment, so what was tested is not what ships |
| Registry | An addressable, stored artifact | A tag moved; the digest is not the one you tested |
| Deployment system | A rollout | Applied successfully while pods never became ready |
| Infrastructure | Running compute | Capacity unavailable; quota exhausted; zone degraded |
| Application | Served requests | Started but failing readiness for a dependency reason (Probes: Readiness, Liveness and Startup) |
| Users | Experience | Everything above healthy; a CDN or DNS layer is serving the old thing |
One identity, all the way through
The single highest-value thing in this model is that one identifier survives every hop. A commit SHA that appears in the build log, the artifact label, the running process and every log line turns most delivery questions into lookups.
Without it, "which version is running" requires archaeology across several systems — and that archaeology happens during incidents, when it is most expensive.
1commit a1b2c3d2 -> build #4821 (records commit a1b2c3d)3 -> image app@sha256:9f3e... (labelled with a1b2c3d)4 -> release r-2026-08-26.3 (names the digest, not a tag)5 -> running process (/version reports a1b2c3d)6 -> every log line (fields: version=a1b2c3d)The point is not the format. It is that at any hop you can ask "which commit is this?" and get an answer without asking a person.
How to do it properly
Most important first.
- Learn to verify each hop directly: which commit built, which digest is in the registry, which digest is running (Tags Versus Digests).
- Make the running version observable from the application itself — a version endpoint or a startup log line closes the biggest gap in the chain.
- When something is wrong, walk the chain from the end you can observe most cheaply rather than from the beginning.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A break in the delivery chain typically affects everything that ships through it, which is every service using that pipeline.
What can go wrong
- A deploy that reports success while the previous version is still serving, because the rollout never completed (A Successful Deploy Is Not Evidence of a Healthy System).
- A registry tag that moved, so "the same version" is not the same bytes (Tags Versus Digests).
- Config applied out of band from the artifact, so the two disagree about what is running.
- "The deploy succeeded, so the new code is running." A successful deploy means the deployment system accepted the request, not that the rollout completed or that instances are healthy.
- "It is the same version." Not unless you compared digests. Tags move (Tags Versus Digests).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can determine, from outside the system, which commit is currently serving production.
- Each hop emits something you can check without asking a person.
- Knowing the chain is what makes rollback precise: you can roll back the artifact, the config or the infrastructure independently, and they are three different actions with three different risks.
- Automate the propagation of identity along the chain — commit SHA into the build, into the artifact label, into the running process, into logs.
- That single thread is what turns "which version is broken" from an investigation into a lookup.
- Instrumenting every hop is work that pays off only during failures, which makes it easy to defer and expensive to lack.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe chain has this shape whether the deployment target is Kubernetes, a PaaS, VMs or serverless — only the names of the last three hops change.
- TOOL-SPECIFICWhere each hop is observable differs by tooling: some CI systems expose the resolved dependency set, some registries expose digest history, some deployment systems record the previous revision. The questions are the same; where you look is not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.