K8s RuntimeKUBERNETES-SPECIFICGENERAL

Apply Is Not Running

Accepted, scheduled, pulled, started, ready and receiving traffic are six different moments, separated in time — so desired state is never instant reality.

What actually happensHow to do it properlyEvidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

Why does kubectl apply return success while nothing new is serving yet, and what is happening in the gap?

The problem

Every stage between storing desired state and serving a request can take time or fail, and the tool that reported success only knows about the first one.

What teams do first

The apply succeeded and the rollout command is running, so the new version is live. Move to the next pipeline step: run the migration, flip the flag, mark the release done.

How it breaks

The apply confirms that the API server validated and persisted a document. At that instant zero new containers exist anywhere in the cluster.

How it breaks in production
  • The apply confirms that the API server validated and persisted a document. At that instant zero new containers exist anywhere in the cluster.
  • A pipeline that proceeds immediately runs its next step against the old version. If that step is a schema contraction, the old version is now broken and the new one has not arrived to replace it (Expand, Migrate, Contract).
  • Image pull time is unbounded from the cluster's point of view — a cold node, a large image or a slow registry stretches the gap arbitrarily (What Image Size Actually Costs).
  • Readiness is not the last stage. Even after a pod reports ready, its address has to reach every load balancer and dataplane that routes to it, and each of those propagates on its own schedule (Service Discovery in Operation).
  • Because the stages are independent, a rollout can be half-done for a long time — both versions serving simultaneously, which is normal and is only safe if you designed for it (Version Coexistence: N and N+1, in Both Directions).
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • The stages are: accepted (validated and stored) → acted on (a controller created pods) → scheduled (a node was chosen and the pod bound) → pulled (the image is present on that node) → started (the container process is running) → ready (the readiness probe passes) → routed (the endpoint is in the dataplane).
  • Each stage is a separate loop owned by a separate component, and each reports its own status. There is no single component that knows the end-to-end state, which is why there is no single command that tells you the truth.
  • The gap is not a defect being worked around. It is what lets the cluster absorb a node dying without an operator: the same asynchrony that delays your deploy is what recovers your workload.
  • Old and new coexist during the whole gap. On a rolling update, some fraction of traffic hits the old version and some the new for the entire duration (Rolling: Two Versions, One Database).
  • Routing propagation is the stage most often forgotten. A pod that has just become ready is not receiving traffic yet, and a pod that has just been deleted may still receive some (Draining: Stopping Without Dropping).

Six stages, six ways to stall

Every stage has its own failure mode and its own evidence. Knowing which stage you are in is most of the diagnosis, and it is why "the deploy is stuck" is not a useful sentence.

From `kubectl apply` to a served request
  1. 1
    Accepted

    API server validates the object and persists desired state.

    fails by Schema error, admission webhook rejection, quota denial — this is the only stage that fails your command.

    evidence The apply returned, and metadata.generation incremented.

  2. 2
    Acted on

    The Deployment controller creates or scales a ReplicaSet, which creates pods.

    fails by Controller down or throttled; observedGeneration stays behind (Reconciliation: The Loop Under Everything).

    evidence observedGeneration matches generation; new pods exist.

  3. 3
    Scheduled

    The scheduler filters and scores nodes, then binds the pod to one.

    fails by No node fits: pods sit Pending with FailedScheduling (The Scheduler, and Why a Pod Is Pending).

    evidence Pod has a nodeName.

  4. 4
    Pulled

    The kubelet pulls the image onto that node if it is not cached.

    fails by ImagePullBackOff — wrong tag, missing pull secret, registry unavailable (Artifact Registries).

    evidence Container state leaves Waiting.

  5. 5
    Started

    The container process runs and begins initialising.

    fails by Crash on startup, missing config or secret, CrashLoopBackOff (Validate at Startup, Fail Clearly).

    evidence Container state Running with a start time.

  6. 6
    Ready

    The readiness probe passes, so the pod is eligible for traffic.

    fails by Probe failing on a dependency, or too aggressive for a slow boot (Probes: Readiness, Liveness and Startup).

    evidence Pod condition Ready is True.

  7. 7
    Routed

    The endpoint is added and every dataplane that routes to the service learns about it.

    fails by Propagation delay, stale client-side caches, external load balancer lag (Service Discovery in Operation).

    evidence A request through the service address reaches the new pod.

Only the first stage can fail your kubectl command. Every other stage fails asynchronously, minutes later, into a status field that nothing is watching unless you made it watch.

A rollout that looked fine

SIMULATEDA reconstruction in the Engineer Atlas incident model, not a measured incident. The relative ordering is the teaching; the timestamps are labels, not durations.

This is the shape of the incident, without the tool names. Nothing here is a bug in Kubernetes; every component did exactly what it was designed to do, on its own schedule.

Deploy plus contracting migration, ordered on the assumption that apply means running
  1. T+0changePipeline applies the new Deployment; kubectl apply returns success
  2. T+0signalReplicaSet created; first new pod is Pending — no node has room for its request
  3. T+0changePipeline proceeds to the next step and runs the migration that drops the old column
  4. T+1signalOld pods, which are the only pods serving, begin returning errors on the dropped column
  5. T+2signalError rate alert fires; dashboards show the deploy annotation and assume the new version is at fault
  6. T+3actionOn-call rolls back the Deployment — which changes nothing, because the new version never served a request
  7. T+4signalSomeone reads pod status and finds every new pod still Pending
  8. T+5recoveryCluster capacity is increased; new pods schedule, start, become ready and serve correctly
  9. AfterrecoveryAction item: the deploy step waits for rollout completion with a timeout, and the contraction moves to a later, separate change

The rollback made things worse in the sense that it consumed the first minutes on the wrong hypothesis. The deploy was never the cause — the ordering assumption was.

changesignalactionrecovery

What each tool actually knows

KUBERNETES-SPECIFICThe signal names are Kubernetes; the structure is universal. On a VM fleet the equivalent row is "the instance is InService in the target group", which likewise does not prove the application is answering correctly.

The reason this gap survives so many pipelines is that every tool in the chain reports honestly about its own stage and says nothing about the rest. Read the middle column as "the strongest claim this signal supports".

SignalWhat it provesWhat it does not prove
kubectl apply exit 0The API server validated and stored desired stateThat any controller has acted, or any pod exists
observedGeneration == generationThe controller has seen this specThat pods scheduled, started or became ready
Pods RunningContainer processes started on their nodesThat the application initialised or can serve
Pod condition ReadyThe readiness probe passedThat routing has been updated to include this pod
rollout status returnedThe control plane considers the rollout completeThat external load balancers, DNS or clients see the new pods
A request returns the new versionThis path, right now, reaches the new codeThat every path does — which is what canary analysis is for

How to do it properly

Most important first.

  • Make the pipeline wait for a signal from the end of the chain, not the beginning. kubectl rollout status with a timeout is a floor, not a ceiling — and it must have a timeout, or a stuck rollout blocks the pipeline forever.
  • Verify the thing you actually care about: that the new version is answering requests. A version endpoint checked through the service beats any control-plane status (A Successful Deploy Is Not Evidence of a Healthy System).
  • Order pipeline steps so that nothing depends on the new version existing until you have evidence it does. Expand-only migrations first, contraction as a separate later change.
  • Assume both versions run at once, and make every change compatible in both directions for the duration of a rollout (Version Coexistence: N and N+1, in Both Directions).
  • Pre-pull or keep images small if pull time is the dominant stage, particularly where nodes are created on demand and every new node starts cold.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

The rolling update contains the deploy itself — old pods keep serving while new ones fail to become ready, so a failed rollout is often invisible to users. What is not contained is anything the pipeline does next on the assumption that the new version is live, which is how a routine deploy becomes a total outage.

What can go wrong

Failure modes, including of the mitigation
  • Pipeline declares success on apply, and the change is later found not to have rolled out at all.
  • rollout status without a timeout, turning a stuck rollout into a hung pipeline and a paged engineer.
  • A migration or a feature-flag flip that assumed the new code was serving, executed against the old code.
  • Health verification that queries the control plane rather than the service, so it passes while the dataplane still routes to nothing.
  • A rollout that never completes and is never noticed, because the old version keeps serving perfectly and nothing alerts on "the new version is not here".
Misreads this invites
  • "Eventually consistent means unreliable." It means the stages are separated in time. The system is highly reliable about converging; it makes no promise about when.
  • "kubectl rollout status returning means users have the new version." It means the control plane considers the rollout complete. Routing, caches and long-lived connections can still be serving the old one (DNS in Production).
  • "If it is slow, something is wrong." Pull, start and readiness times are real work. The question is whether the stage is progressing, not whether it is fast.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • Requests through the service return the new version identifier — checked from outside the cluster, on the path users take.
  • kubectl rollout status returned within its timeout, and updated / ready / available replica counts all match the desired count.
  • Deployment condition Available is True and the endpoint list for the service contains the new pods.
  • Error rate and latency on the canary or new replicas are compared against the old ones, not against an absolute threshold (Canary Analysis: Compared Against What?).
How you get back
  • While the gap is open, rollback is nearly free: the old version is still running and still serving. Restore the previous desired state and the new pods are removed before they ever took traffic.
  • Once the rollout completes, rollback is a second rollout with the same gap and the same stages — it is not instant either, and planning for that matters when you are choosing between rolling back and rolling forward (Roll Forward: When Going Back Is the Harder Option).
What to automate, and what stays human
  • Automate the wait and the verification together. A deploy step that applies and returns is not a deploy step; it is half of one.
  • Automate a hard timeout with an explicit failure. A rollout that has not converged in the time you allowed is a decision point for a human, not something to keep waiting on.
What this costs
  • Waiting for real readiness makes pipelines slower and turns some previously "successful" deploys into failures. That is the correct trade, and it will be unpopular the first week.
  • Designing every change to be compatible with the previous version costs real engineering effort on changes that would otherwise be one commit (Expand, Migrate, Contract).

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • KUBERNETES-SPECIFICThe six-stage separation and the apply / running distinction are Kubernetes. A PaaS deploy command usually blocks until the new instances pass its health check and reports a single success or failure, hiding the stages — and giving you less visibility when one of them is the problem. A VM autoscaling group has the same gap with different names: launch, boot, user-data, health check, target-group registration.
  • GENERALThe underlying property — "the change is accepted long before it is in effect" — holds for DNS, CDN invalidation, config distribution and load balancer registration on every platform.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendhealth-checksbackend-on-kubernetes
Cloudkubernetes-objectsstartup-time-and-cold-start
Observability & Performancedeployment-markers
Domains that do not exist yet
  • — Distributed Systems — why a control plane that converges eventually is a deliberate design choice rather than a compromise.