What Can Fail Between Commit and User
A catalogue of delivery failures, arranged by where in the chain they happen and what they look like from the outside.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When a change does not reach users correctly, what are the actual candidate causes?
Delivery failures present with generic symptoms — "the deploy is broken", "the new version is not live" — which describe the observation rather than narrowing the cause.
When a deploy fails, look at the deployment system's logs. That is where deploys happen, so that is where the failure will be.
The deployment system is one hop of ten. It reports faithfully on its own step and knows nothing about the nine others.
- The deployment system is one hop of ten. It reports faithfully on its own step and knows nothing about the nine others.
- Several of the most common failures make the deployment system report success: a wrong artifact deployed correctly, a config that did not change, an image that pulled from cache.
- Some failures happen before the pipeline ran at all — a check that does not exist cannot fail.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Delivery failures fall into a small number of families, and each family has a characteristic signature: what the pipeline says, what the platform says, and what users see.
- The most confusing ones are those where every system reports success. Those are almost always identity failures — the right process is running the wrong bytes, or the right bytes with the wrong config.
The families, by signature
Read the middle columns as what you would observe. The point is that several families are indistinguishable from the application side, and only the identity checks separate them.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Broken build | Pipeline red; nothing deployed | Compilation or check failure | Fix forward; the safest failure there is, because nothing shipped |
| Bad dependency | Pipeline green; behaviour changed with no code change | Unpinned dependency resolved to a new version | Pin and rebuild (Dependency Pinning) |
| Wrong artifact | Everything reports success; old behaviour persists | A tag moved, or a cached image was used | Deploy by digest, not tag (Tags Versus Digests) |
| Bad config | Immediate failure on start, or wrong behaviour with correct code | Config changed independently of the artifact | Validate at startup; version config with the release (Validate at Startup, Fail Clearly) |
| Missing secret | Crash loop, often with an unhelpful error | Secret not present in the target environment | Fail loudly at startup rather than lazily at first use (When Secrets Fail) |
| Failed migration | Writes failing; old version erroring | Schema changed incompatibly with running code | Expand/migrate/contract; never contract during rollout (Expand, Migrate, Contract) |
| Unavailable infrastructure | Pods pending; instances not launching | Quota, capacity or zone issue | Check platform events before debugging the app (The Scheduler, and Why a Pod Is Pending) |
| Failed rollout | Apply succeeded; new version never became ready | Readiness probe failing, often for a dependency reason | Read probe failures, not application logs first (Probes: Readiness, Liveness and Startup) |
| Partial deployment | Intermittent behaviour; some requests old, some new | Rollout stalled mid-way; both versions serving | Verify version distribution; this is normal briefly and a problem if sustained (Version Coexistence: N and N+1, in Both Directions) |
| Bad health check | Restart loop; capacity falling | Liveness depends on a failing dependency | Liveness must test the process, not the world (Probes: Readiness, Liveness and Startup) |
| Capacity shortage | Latency rising; errors under load | Fewer healthy instances than the traffic needs | Restore capacity first, diagnose after (Headroom) |
| Monitoring blind spot | Users report a problem dashboards do not show | No signal exists for this failure mode | Add the signal as the action item, not "watch more closely" (An Alert Should Demand Action) |
The three checks that separate most of them
Four of these families — wrong artifact, bad config, failed rollout, partial deployment — look identical from the application's behaviour and are separated by three cheap questions.
Getting these onto one page is one of the highest-value pieces of operational tooling a team can build, because it converts the most confusing incident class into a lookup.
- 1What version is running?
Ask the application, not the deployment system.
fails by No version endpoint, so the answer requires archaeology.
evidence Reported commit matches the release record.
- 2What digest is deployed?
Compare the running image digest to the one the release names.
fails by Deploying by tag, so the question has no stable answer (Tags Versus Digests).
evidence Digest matches exactly.
- 3What config did it load?
Ask the process which config version it started with.
fails by Config applied out of band and never surfaced.
evidence Config version matches the release record (The Release Manifest).
If all three agree and behaviour is still wrong, the problem is genuinely in the code or in a dependency — which is a much smaller search space than where you started.
How to do it properly
Most important first.
- Work the chain rather than the logs of one system (From Developer to Users).
- Start from the cheapest observation that discriminates: what version does the running application say it is?
- Ask what changed, in all four change categories — code, config, infrastructure, dependencies (Change Correlation).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Depends entirely on the rollout strategy in use; the same failure is one percent of traffic with a canary and everyone without one (Reducing Blast Radius).
What can go wrong
- Assuming a green pipeline means the artifact is correct. Green means the checks that exist passed.
- Assuming a successful apply means a completed rollout.
- Assuming the failure is in the newest change, when a config or infrastructure change landed at the same time.
- "The pipeline is green so the code is fine." Green means the checks that exist passed. A missing check cannot fail (Required Checks).
- "Nothing changed." Something almost always changed — if not code, then config, infrastructure, a dependency, or traffic.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The running version reported by the application matches the release record.
- The digest in the registry matches the digest the release names.
- The config the process loaded matches the config version the release names.
- Which rollback applies depends on the family: an artifact problem rolls back the artifact; a config problem rolls back the config; an infrastructure problem may need neither. Rolling back the wrong layer wastes the most valuable minutes of an incident.
- Automate the three identity checks above and surface them on one page. Most delivery incidents are resolved by whichever of the three disagrees.
- Do not automate remediation of an unknown failure — an automated rollback triggered by an unreliable signal is its own incident.
- Checking every hop costs time during an incident. In practice you learn which two or three hops your organisation fails at most and check those first — at the cost of occasionally missing an unusual one.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe families hold across toolchains. Where each is observable varies, which is why the lesson is organised by symptom rather than by tool.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.