Horizontal Pod Autoscaling
One concrete implementation of the control loop — how Kubernetes does it, and what a VM autoscaling group or a serverless platform does instead.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does the Kubernetes implementation of autoscaling work, and what does it assume that other platforms do not?
Autoscaling is usually learned through one implementation, so its accidental properties get mistaken for how autoscaling works everywhere.
Add a HorizontalPodAutoscaler targeting CPU utilisation and the workload scales. It is a few lines of YAML and it is the documented way to do it.
Utilisation targets are computed against the pod's CPU request, not against the node or against any absolute capacity. A workload with no request set has no denominator, so utilisation-based scaling does not work at all (Requests and Limits).
- Utilisation targets are computed against the pod's CPU request, not against the node or against any absolute capacity. A workload with no request set has no denominator, so utilisation-based scaling does not work at all (Requests and Limits).
- That coupling means changing a request silently retunes the autoscaler. A right-sizing exercise can make an autoscaler far more or far less eager without anyone touching the scaling policy (Overprovisioning).
- Adding replicas only helps if there are nodes to place them on. When there are not, the pods stay pending and the workload does not scale until a separate cluster autoscaler adds a node — a much slower loop (The Scheduler, and Why a Pod Is Pending).
- The controller works from an average across pods, so a workload with skewed load — a hot shard, sticky sessions — can have saturated pods while the average looks comfortable.
- Scaling replicas raises the fleet's connection count and downstream call rate exactly like any other horizontal scaling, and the manifest gives no hint of that (The Connection Budget).
- A horizontal autoscaler and a vertical one on the same workload will fight, because each changes the input the other is measuring.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The Kubernetes horizontal autoscaler is the same control loop as everywhere else: it reads a metric, compares it to a target, and sets the replica count of a scalable workload.
- For resource metrics the target is a percentage of the pod's request, averaged across ready pods. Desired replicas is derived from the ratio of the current value to the target, so the request value is part of the policy whether you intended it or not.
- Metrics arrive from a metrics pipeline that samples periodically, so the controller acts on a value that is already somewhat stale — the collection delay of the general delay budget (Autoscaling).
- Scaling behaviour is damped by a stabilisation window and by policies that bound how fast the replica count may change, configured separately for scale-up and scale-down.
- The replica count is a desired state; the scheduler still has to place the new pods. Autoscaling and scheduling are two different loops, and the second is where "it scaled but nothing happened" comes from (Reconciliation: The Loop Under Everything).
- A VM autoscaling group does the equivalent with instances: a target-tracking or step policy on a cloud metric, an instance warm-up period during which the new instance is excluded from metrics, and a cooldown before the next action. The unit is bigger and slower and there is no scheduler in between.
- A serverless platform does not expose a replica count at all. It creates execution environments in response to concurrency and removes them when idle, with the scaling policy owned by the provider — you configure concurrency limits and provisioned warm capacity instead (Scale to Zero).
The same loop on three platforms
Read this table from the last column backwards. The failure mode is what actually differs between platforms; the concept is the same everywhere.
| Aspect | Kubernetes horizontal autoscaler | VM autoscaling group | Serverless platform |
|---|---|---|---|
| Unit added | A pod replica | An instance | An execution environment |
| Utilisation denominator | The pod's resource request | The instance's capacity | Not exposed; concurrency is the unit |
| Placement | The scheduler, as a separate loop | The provider places the instance | Provider-managed entirely |
| Warm-up handling | Readiness probes gate traffic (Probes: Readiness, Liveness and Startup) | An instance warm-up period excludes it from metrics | First request pays the cold start (Scale to Zero) |
| Damping control | Stabilisation window and change-rate policies | Cooldown between scaling activities | Provider policy; you set concurrency limits |
| Typical lag driver | Image pull, then node addition if none is free | Instance boot time | Environment initialisation |
| Characteristic failure | Replicas increase, pods stay pending | Boot too slow for the load shape | Cold starts on the latency-sensitive path |
What the manifest actually says
The object is small, and every value in it is a decision. The two that matter most are the two that look like boilerplate: the minimum, which is the standing capacity that carries load during the scaling delay, and the maximum, which is the only bound on scaling into a shared dependency.
1apiVersion: autoscaling/v22kind: HorizontalPodAutoscaler3metadata:4 name: checkout5spec:6 scaleTargetRef:7 apiVersion: apps/v18 kind: Deployment9 name: checkout10 minReplicas: 6 # standing capacity: must cover the scaling11 # delay plus the failure reserve12 maxReplicas: 40 # derived from the downstream connection budget,13 # NOT from what the cluster could fit14 metrics:15 - type: Resource16 resource:17 name: cpu18 target:19 type: Utilization20 averageUtilization: 70 # a percentage of the pod REQUEST,21 # averaged across ready pods22 behavior:23 scaleDown:24 stabilizationWindowSeconds: 300 # asymmetric on purpose:25 policies: # slow to give capacity back26 - type: Percent27 value: 1028 periodSeconds: 6029 scaleUp:30 stabilizationWindowSeconds: 0 # fast to take itThe replica numbers and the utilisation target are placeholders standing in for values you must derive for your own service — they are not recommendations. What is worth copying is the reasoning in the comments: minimum from the delay budget, maximum from the downstream limit, and deliberate asymmetry between scale-up and scale-down.
Scaled, but not scaled
The most confusing Kubernetes autoscaling failure is the one where every object reports success. The autoscaler set the replica count it wanted, the deployment accepted it, and no capacity appeared — because the two loops involved are separate and only one of them ran.
- 1Metrics pipeline
Samples pod resource usage and makes it queryable.
fails by Metrics unavailable or stale, so the controller cannot compute anything.
evidence The autoscaler reporting an unknown or stale metric value.
- 2Autoscaler evaluation
Compares the averaged value against the target and computes desired replicas.
fails by No request set, so utilisation has no denominator and the target is meaningless (Requests and Limits).
evidence Current and target values in the autoscaler status.
- 3Workload update
The replica count on the deployment is changed.
fails by Rarely — this step almost always succeeds, which is why the failure feels invisible.
evidence Replica count in the workload spec.
- 4Controller creates pods
New pods are created to match the desired count (Reconciliation: The Loop Under Everything).
fails by Quota on the namespace blocks creation.
evidence Pod count created versus desired.
- 5Scheduler places pods
Finds a node with enough unreserved capacity for the request.
fails by No node has room; pods stay pending until a node is added (The Scheduler, and Why a Pod Is Pending).
evidence Pending pod count and scheduling events.
- 6Cluster autoscaler adds a node
A separate, slower loop provisions a node when pods cannot be placed.
fails by Node provisioning takes minutes, or hits a quota or zone capacity limit.
evidence Time from pending to scheduled.
- 7Pod becomes ready
Readiness passes and the service begins routing to it (Probes: Readiness, Liveness and Startup).
fails by Readiness passes before warm-up, so traffic reaches a cold pod.
evidence Per-pod latency in the first minutes after start.
The gap between step three and step five is where "we scaled and nothing happened" lives. Replica count is a desired state; running, ready, warm pods are capacity, and only the last of those serves anyone.
How to do it properly
Most important first.
- Set resource requests deliberately, from measurement, before enabling utilisation-based scaling. The request is half the policy (Requests and Limits).
- Treat any change to requests as a change to the autoscaling policy, and re-check the scaling behaviour afterwards.
- Set minimum replicas from the delay budget and the failure reserve, never from the smallest number that serves quiet-hour traffic (Headroom).
- Set maximum replicas from what the downstream can take — connection budget, dependency quota — rather than from cluster capacity.
- Check that the cluster can actually place the maximum. A ceiling the scheduler cannot satisfy is a number, not capacity.
- Prefer a signal that reflects the constraint over CPU utilisation just because it is the default (Choosing the Scaling Signal).
- Do not run a horizontal and a vertical autoscaler on the same resource of the same workload.
- Configure scale-down stabilisation explicitly; the default behaviour is a policy decision made by someone who did not know your traffic.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by minimum and maximum replicas, and by the cluster's own capacity — which contains it by failing to place pods rather than by any deliberate design.
What can go wrong
- Utilisation-based scaling on a workload with no CPU request, which does not scale and reports no error anybody looks at.
- Replicas increased while pods remain pending for lack of nodes, so the workload is nominally scaled and actually is not.
- A CPU limit set close to the request, so the pod throttles before utilisation reaches the target and the autoscaler never triggers (CPU Throttling: The Latency With No Error).
- Scale-down removing pods mid-request because termination handling is missing (Graceful Shutdown).
- Skewed load across replicas hidden by the average, leaving some pods saturated at a comfortable mean.
- The mitigation failing: a conservative stabilisation window that also prevents the workload from releasing capacity, so the fleet ratchets up over a week.
- "HPA is what autoscaling means." It is one implementation with specific coupling to requests, to the scheduler and to a metrics pipeline. A VM group and a serverless platform solve the same problem with different failure modes (Do You Need Kubernetes?).
- "Utilisation target is a percentage of the node." It is a percentage of the pod's request. Two workloads with the same target and different requests behave completely differently.
- "Setting maximum replicas high is harmless." The maximum is the only thing standing between a runaway loop and your database.
- "It scaled the replicas, so it added capacity." Only if the scheduler placed them. Replica count is desired state, not running capacity (Reconciliation: The Loop Under Everything).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The autoscaler reporting current and target metric values, and the reason for its last decision.
- Replica count plotted against pending pod count — the pair that distinguishes "did not scale" from "scaled and could not be placed".
- Per-pod utilisation distribution, not just the average, to reveal skew.
- CPU throttling counters, since throttling suppresses the very signal the autoscaler reads.
- Downstream connection count plotted against replica count.
- The fastest rollback is to set minimum and maximum replicas to the same known-good value, which pins the workload and removes the loop from the incident.
- Autoscaler objects are declarative, so reverting the manifest reverts the policy — provided the manifest is the source of truth and nobody has been editing live objects (Drift).
- Reverting a request change reverts the autoscaling behaviour with it, which is a rollback interaction worth knowing about before it surprises you.
- Automate the policy as part of the workload manifest, reviewed and deployed with the service rather than configured by hand (Infrastructure as Code).
- Automate an alert on maximum replicas reached and on pods pending, both of which are silent by default.
- Keep minimum and maximum human. They encode a reliability purchase and a downstream limit, neither of which is visible from inside the cluster.
- Coupling the target to the request keeps the policy portable across pod sizes and makes right-sizing and autoscaling tuning inseparable.
- Averaging across pods is simple and robust and hides skew, which matters most for exactly the workloads where load is uneven.
- Damping the loop stabilises replica count and slows the release of capacity, which shows up on the bill rather than in an incident.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICRequests as the utilisation denominator, pod averaging, the split between the autoscaling loop and the scheduler, and stabilisation windows are Kubernetes mechanics. A VM autoscaling group tracks a cloud metric against instance capacity with an instance warm-up and a cooldown, and has no scheduler in between; a serverless platform scales execution environments on concurrency with no replica count exposed at all.
- GENERALThe underlying loop — signal, target, policy, actuator, lag — is identical on all three. What is portable is the reasoning; what is not is any specific field, default or unit.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — verifying that a declared desired state actually became running capacity.