The question this answers
Once a workload is a hundred containers spread across twenty machines, who decides what runs where — and who puts it back when it dies at 03:00?
The product is eleven services: an API, a web frontend, four background workers, a scheduler, two internal gRPC services and two batch jobs. Each runs several replicas for redundancy, each releases independently, and none of them may be down while a machine is rebooted for a kernel patch.
A single place where you declare what should be running, and a machine that continuously makes reality match it — including choosing hosts, restarting failures, replacing whole machines and shifting traffic during a release.
Derive it from the problem list, not from the product
Start with three containers on one machine. You start them by hand, you know each one's port, you restart the one that died, and when you deploy you stop the old one and start the new one. Every operational job is a command a human types. Nothing here needs orchestration, and adding it would be strictly worse.
Now make it a hundred containers on twenty machines. Nothing conceptually new happened — but every one of those jobs changed shape. *Which machine* has enough free memory for this container is now an arithmetic problem across twenty hosts, and the answer changes every minute. *Restart the one that died* is now a question of noticing that it died, on a machine nobody is watching, at an hour nobody is awake. *Deploy* is now a hundred coordinated stop/start pairs where a mistake in the middle leaves half the fleet on the old version. *Where do I find the payments service* is no longer "port 8081 on this box" but "one of seven addresses that changes whenever a container is replaced".
Orchestration is the name for the system that answers those questions automatically. The word is worth taking literally: nobody is playing a new instrument, someone is deciding who plays when. The scheduler, the health checks, the rollout controller and the service registry each exist because one item in the list below stopped being answerable by a person.
| Job | At 3 containers on 1 machine | At 100 containers on 20 machines |
|---|---|---|
| Placement | There is one machine. It runs there. | Bin-packing against free CPU and memory on twenty hosts, recomputed on every change. |
| Restart | You notice and type the command again. | Something must watch every container on every host and replace failures without a human. |
| Health | It responds when you curl it. | Every replica needs a probe, and "started" has to be distinguished from "ready to serve". |
| Networking | localhost and a port you remember. | Every container gets an address that changes on every replacement; callers cannot hard-code any of them. |
| Rollout | Stop old, start new, ten seconds of downtime nobody notices. | A hundred replicas replaced in a controlled order, with a way to stop and reverse halfway. |
| Machine loss | You are down until you fix it. | One host of twenty dies; its containers must reappear elsewhere before anyone files a ticket. |
If you refuse to adopt one, you will write one
Teams that decline orchestration at this scale do not avoid the problem; they solve it incrementally in shell. The progression is so consistent it is almost a law. It begins as a deploy script, acquires a host list, then a health check, then a retry, then a lock so two engineers cannot deploy at once, then a table of which container is on which host — and at that point you are maintaining a scheduler with no tests, no leader election and one author.
This is the honest argument for adopting an orchestrator: not that it is elegant, but that at a certain scale the *work does not go away*, and a system built by thousands of people is usually a better place to put it than a file called deploy.sh that only one person understands. The equally honest counterargument is that most systems never reach that scale, which is what Kubernetes Is Not Always Needed is for.
- The failure this script has already accepted: a mid-loop abort leaves hosts 1–9 on the new image and 10–20 on the old one, with no record of which is which.
- It has no concept of desired state, so it cannot answer "is the fleet currently correct?" — only "did the last run finish?".
- It reacts to a deploy, never to a machine dying at 03:00, which is the failure that actually pages someone.
#!/usr/bin/env bash
HOSTS=(app-01 app-02 ... app-20) # hand-maintained; app-07 was rebuilt last week
for h in "${HOSTS[@]}"; do
ssh "$h" "docker pull registry/api:$TAG" || echo "WARN pull failed on $h" # ignored
ssh "$h" "docker stop api && docker run -d --name api registry/api:$TAG"
sleep 5 # "long enough" — chosen in 2023, never revisited
curl -sf "http://$h:8080/health" || echo "WARN $h unhealthy" # also ignored
done
# TODO: stop on first failure # 14 months old
# TODO: what if two people deploy # 11 months old
# TODO: memory-aware placement # abandonedWhat an orchestrator is, structurally
Every orchestrator — Kubernetes, Nomad, ECS, Swarm, an internal one — has the same three parts. A declared desired state ("ten replicas of api:v7, each needing 512 MiB"), a control plane that stores it and decides how to satisfy it, and an agent on every machine that starts and stops containers and reports what is actually running. The loop between the last two is the entire idea; Kubernetes: Why It Exists derives it in detail.
The part teams underestimate is that the control plane is itself a distributed system you now operate. It needs to be highly available, it needs an upgrade path, it holds credentials for every workload, and when it is unhealthy your ability to *change* anything disappears — usually while the workloads themselves keep running, which makes the outage confusingly quiet. That is the real price on the ticket, and it is why the decision belongs in No Cargo-Cult Infrastructure rather than in a default.
Key points
- Orchestration is derived from six jobs — placement, restart, health, networking, rollout, machine loss — that stop being human-scale somewhere between one machine and twenty.
- The work does not disappear if you decline an orchestrator; it reappears as a deploy script that slowly becomes an undesigned scheduler.
- Every orchestrator is the same three parts: declared desired state, a control plane, and a per-machine agent reporting what is actually running.
- The control plane is a distributed system you now operate, and its outages are quiet: running workloads survive, but you lose the ability to change anything.
- Nothing in this lesson argues that your system has reached that scale. Most have not.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • You declare desired state — how many replicas of which image, with which resource needs and which health probes.
- • The control plane persists that declaration and computes the difference between it and what the agents report is running.
- • A scheduler assigns each unplaced workload to a machine with enough free capacity that satisfies its constraints.
- • The agent on that machine pulls the image and starts the container through the container runtime, then reports status back.
- • Controllers watch the gap continuously and act on it: a missing replica is created, a surplus one is deleted, a failed machine's workloads are rescheduled elsewhere.
- • The control plane itself: availability, backups of its datastore, and a version upgrade cadence you cannot skip.
- • The machines underneath — an orchestrator schedules onto nodes, it does not patch, size or replace them for you unless you also adopt something that does.
- • The declarations: every resource request, probe and replica count is a number a human chose, and wrong numbers fail in the ways Requests vs Limits: Two Numbers That Do Different Jobs describes.
- • The images: an orchestrator will faithfully run a broken image forever. It has no opinion about your build.
- • Control plane unreachable: running workloads keep serving, but deploys, scaling and self-healing all silently stop. The dashboard looks fine and nothing is being fixed.
- • A cluster with no spare capacity: replacements for a dead machine's workloads sit unplaced, and the fleet quietly runs below its declared replica count.
- • Declared state that is wrong: the orchestrator enforces the mistake with perfect discipline, which is why a bad manifest scales an incident instead of containing it.
- • Operator drift: someone changes a running workload by hand, the controller reverts it, and the fix disappears without explanation.
- • Node count scales well; what runs out first is usually control-plane datastore write throughput, driven by object count and churn rather than by traffic.
- • Scheduling latency grows with the number of pending workloads and the complexity of their placement constraints, not with request volume.
- • The human dimension runs out before the technical one: cluster count, namespace sprawl and manifest volume outgrow a small team long before the scheduler does.
- • The control plane holds the credentials and configuration of every workload in the cluster; compromising it is compromising the platform, not one service.
- • Every agent must authenticate to the control plane, and every workload should have an identity distinct from the machine it happens to land on — see Human vs Workload Identity.
- • Multi-tenancy is a deliberate design, not a default: two teams in one cluster share a network, a scheduler and, unless configured otherwise, a lot of read access.
- • The trust boundary moves from "this machine runs our app" to "this cluster runs everything", which is a much larger blast radius to reason about.
- • A control plane is a fixed monthly cost whether it manages three workloads or three hundred — the per-workload cost is therefore terrible at small scale and good at large scale.
- • Nodes are the variable meter, and utilization is the lever: bin-packing is exactly the mechanism that turns twenty half-empty machines into twelve full ones.
- • The largest real cost is rarely on the invoice. It is the engineering time to run, upgrade and debug the platform, which is why Scoring Operational Complexity treats it as a first-class number.
- • Declared replicas versus ready replicas, per workload — the single most useful health signal in an orchestrated system.
- • Pending or unplaced workloads, which is the leading indicator of a capacity or constraint problem.
- • Control-plane API latency and error rate; degradation here shows up as deploys hanging rather than as user-facing errors.
- • The signal that lies: node CPU and memory graphs. A cluster can look comfortably utilized while individual workloads are being throttled or evicted — see OOM Kills and CPU Throttling.
- • A single machine with a process supervisor and a compose file. For one service and a few containers this is correct, and it is the answer far more often than the industry admits.
- • A managed container service that runs containers for you without exposing a cluster — you get placement, restarts and rollouts without operating a control plane.
- • A platform-as-a-service: push a repository, get a running, load-balanced, auto-restarted app. It removes the entire problem list at the cost of flexibility.
- • Two or three VMs behind a load balancer with an immutable image per release. Boring, cheap, and survives a machine loss — see Mutable Servers and Immutable Images.
- • Buys automatic placement, healing and rollout; costs a distributed control plane that your team must run, upgrade and debug.
- • Buys higher machine utilization through bin-packing; costs a new class of failure where workloads interfere with each other on a shared node.
- • Buys a uniform deployment interface across every service; costs a large vocabulary that every engineer must now learn before they can ship.
- • Buys portability of the abstraction; costs real portability, because the networking, storage and identity integrations underneath are provider-shaped.
What people believe, and what is true
Orchestration is what you use to run containers.
A container runtime runs containers. Orchestration decides which machine runs which container and puts it back when it dies — a scheduling and reconciliation problem, not a packaging one.
Adopting an orchestrator removes operational work.
It moves it. You stop restarting containers by hand and start operating a control plane, a node pool, a network layer and a policy model.
You need orchestration once you have containers.
Containers and orchestration are independent decisions. A container on one VM under a supervisor is a complete, defensible production setup.