Scale to Zero
Running nothing when there is nothing to do — and paying for it with a cold start on the next request.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
When is it right for a service to have no capacity running at all?
Most workloads are idle most of the time, and capacity that is provisioned while idle is paid for while idle.
Scale everything to zero when idle. Nobody is using it, so nothing should be running, and the platform will start it again on demand.
The first request after idle pays for everything: allocating the environment, pulling or loading the image, starting the runtime, initialising the application, connecting pools and filling caches (Autoscaling).
- The first request after idle pays for everything: allocating the environment, pulling or loading the image, starting the runtime, initialising the application, connecting pools and filling caches (Autoscaling).
- That cost lands on a real user, and it lands on an unpredictable subset of them, so it shows up as a tail-latency problem rather than an average one.
- Connection-heavy services re-establish everything on each cold start. A burst of cold starts produces a burst of new database connections, which is the classic way scale-to-zero takes down a database (The Connection Budget).
- Runtimes that rely on warm-up — JIT compilation, lazy loading, populated caches — are slow for a while after start even once they are healthy, so readiness passing does not mean ready.
- Anything holding local state, a long-lived connection, or an in-memory schedule cannot go to zero without losing it (Why Stateful Workloads Are Harder).
- Scaling to zero also removes your ability to observe the service. There is nothing running to emit metrics, so "no traffic" and "broken" look the same.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Scale to zero is the extreme of the general trade: standing capacity costs money and removes lag; zero standing capacity costs lag and removes the money.
- The cost is not one number. It decomposes into environment allocation, image or code load, process start, application initialisation and warm-up — and which term dominates varies enormously by runtime and by image size (What Image Size Actually Costs).
- The cost is paid per cold start, not per idle period. A workload receiving sparse, scattered requests can pay it constantly, which is the case where scale-to-zero is worst and looks best on a utilisation graph.
- A scale-to-floor alternative exists on every platform: keep a small number of warm units, accept a small standing cost, and remove the cold start from the common path. Serverless platforms sell this as pre-warmed or provisioned capacity; a container platform expresses it as a non-zero minimum (Horizontal Pod Autoscaling).
- Faking traffic to keep instances warm is the anti-pattern version of the same idea. It pays for the capacity anyway, adds synthetic load to your telemetry, and often fails to keep the right instances warm.
- The right unit of decision is the request path, not the service. Interactive paths and background paths can have different floors even in the same system.
What the first request pays for
Cold start is treated as a single property of a platform, and it is really a sum of phases you have varying control over. Measuring the breakdown usually reveals that one term dominates, and that term is frequently something you chose.
- 1Environment allocation
The platform finds or creates a slot to run in.
fails by Contention at the platform level during a burst, which you cannot influence.
evidence Platform-reported initialisation duration, where exposed.
- 2Code or image load
The image is pulled or the deployment package is fetched and unpacked.
fails by A large image with no warm cache dominates everything else (What Image Size Actually Costs).
evidence Image pull duration on a node that did not have the layer cached.
- 3Process start
The runtime starts and the application entry point runs.
fails by Heavy static initialisation and eager dependency loading at import time.
evidence Time from process start to the application's first log line.
- 4Application initialisation
Configuration is loaded and validated, clients are constructed, pools are opened.
fails by Eagerly connecting to everything, which also produces the connection burst (Validate at Startup, Fail Clearly).
evidence Time from first log line to readiness, and connections opened per start.
- 5Warm-up
Caches fill and the runtime reaches steady-state performance.
fails by Readiness passes here, so traffic arrives while the instance is still slow (Probes: Readiness, Liveness and Startup).
evidence Per-instance latency over the first minutes, compared with a warm instance.
Only the first phase is genuinely the platform's. The other four are consequences of your image, your dependencies and your initialisation code, which is why cold start is more tractable than its reputation suggests.
Where zero is right, and where a floor is
The decision is per request path, not per company or per platform. A system can reasonably run its interactive API on a warm floor and its report generator at zero.
| Workload | Idle fraction | Latency sensitivity | Sensible setting |
|---|---|---|---|
| Interactive user-facing API | Low | High | Warm floor sized for the scaling delay |
| Internal admin tool | Very high | Low — users tolerate a pause | Zero |
| Preview or per-branch environment | Very high | Low | Zero, with an expiry as well (Preview Environments) |
| Scheduled batch job | High between runs | None | Zero; start on schedule |
| Queue consumer with steady arrivals | Low | Bounded by backlog age | Small floor; scale on backlog (Queue-Based Autoscaling) |
| Queue consumer with sparse arrivals | High | Tolerant of delay | Zero, if a start per batch is acceptable |
| Connection-heavy service | Any | Any | Avoid zero, or front the database with a pooler |
| Anything holding local state | Any | Any | Not a candidate (Why Stateful Workloads Are Harder) |
Three ways to pay
There is no option that avoids both the idle cost and the cold start. What there is, is a middle option that most teams skip on the way from one extreme to the other.
A service is idle most of the day and sees short bursts of real use. What runs during the idle period?
when The path tolerates a start-up pause, and starts are infrequent relative to the idle time.
cost Every burst begins with cold starts, and connection-heavy services propagate that downstream.
when The path is latency-sensitive but the peak is far above the floor.
cost A continuous cost, small relative to peak, that has to be justified once.
when Bursts are unpredictable, instant and business-critical.
cost The peak cost, paid all day (Overprovisioning when nobody re-checks whether it is still needed).
when The bursts are predictable — a business-hours pattern, a known job, a campaign.
cost Requires the prediction to hold; an unscheduled burst still finds you cold.
when Rarely defensible — only where the platform offers no warm floor at all.
cost Pays for capacity anyway, pollutes telemetry, and warms an arbitrary subset of instances.
How to do it properly
Most important first.
- Decide per path, from the latency requirement. Anything a user waits on synchronously is a poor candidate for zero.
- Use a small warm floor rather than zero for anything latency-sensitive, and treat the floor as a purchase you can justify (Headroom).
- Measure your actual cold start, broken down by phase, before deciding. It is usually dominated by one term and that term is often fixable.
- Shrink the start path deliberately: smaller images, less initialisation work, lazy connections, deferred non-essential setup (What Image Size Actually Costs, Multi-Stage Builds).
- Bound the connection consequence of a cold-start burst — a proxy or pooler in front of the database is the standard answer (The Connection Budget).
- Keep synthetic keep-warm traffic out of your production telemetry if you use it at all, and prefer a platform-native warm floor to inventing one.
- Alert on absence differently for a scale-to-zero workload, because zero instances is normal and indistinguishable from broken without an external check (An Alert Should Demand Action).
- Make cold-start latency visible as its own metric rather than letting it hide inside an aggregate (Tail Latency: Why p50 Being Fine Does Not Help in Performance).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by the fact that cold starts affect the requests that arrive during them — unless the resulting connection burst reaches a shared database, at which point it is contained by nothing.
What can go wrong
- Cold starts concentrated at the start of a traffic ramp, so the first users of every peak get the worst experience.
- A cold-start burst opening connections faster than the database accepts them, turning an idle-cost optimisation into an outage.
- Readiness passing before warm-up completes, sending traffic to instances that are healthy and slow (Probes: Readiness, Liveness and Startup).
- A scheduled job that assumed something was always running — an in-memory timer, a cached lease, a long-poll subscription — silently stopping (Cron Jobs in Production).
- Monitoring that cannot distinguish scaled-to-zero from failed, so an outage is invisible until someone complains.
- The mitigation failing: keep-warm traffic that keeps one instance warm while real traffic lands on cold ones, so the cost is paid and the benefit is not.
- "Serverless means no cold starts because the provider handles it." The provider handles the scaling. The cold start is the physics of starting a process, and it is exposed to your users as latency (Serverless Trade-offs in Cloud).
- "Cold starts only affect the first request." They affect the first request to each new environment, which during a ramp means many requests, spread across many users.
- "Keeping instances warm with synthetic traffic is a good workaround." You are paying for the capacity, adding noise to your metrics, and getting less reliability than a supported warm floor would give you.
- "Scale to zero saves the most money." It saves the most on idle. If the workload is not actually idle much, a warm floor costs little more and removes the whole problem.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Cold-start latency measured per phase, and the count of cold starts per hour.
- The share of user requests that hit a cold start — the number that determines whether this is a real problem or an academic one.
- Database connection rate during a cold-start burst.
- An external check proving the service responds, independent of whether anything is running.
- Idle-hours cost before and after, so the saving is a number rather than an assumption (Cost Drivers).
- Setting a non-zero floor is the rollback, and it is usually a single configuration value that takes effect quickly (A Config Change Is a Production Change).
- It is not instant: the floor has to actually start before it helps, so during an incident the first minutes still pay cold starts.
- If cold starts caused a downstream connection storm, restoring the floor does not immediately settle the dependency — connection churn has its own recovery time.
- Automate the floor as a configuration value tied to the path's latency requirement, deployed with the service.
- Automate scheduled pre-warming for known traffic patterns, which converts an unpredictable cold start into a predictable one at a chosen moment.
- Keep the choice of which paths may go to zero human, because it is a user-experience decision rather than a utilisation one.
- Zero is the cheapest idle state and the most expensive first request. There is no configuration that avoids both.
- A warm floor removes most of the cold-start pain for a fraction of the peak cost, and is the right answer far more often than either extreme.
- Optimising start-up time is real engineering work that competes with features, and it pays off in scaling lag as well as in cold starts — which makes it better value than it first appears (Autoscaling).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- PLATFORM-SPECIFICServerless platforms scale to zero by default and expose a pre-warmed capacity setting to opt out; container platforms need an add-on or a request-aware proxy to reach zero at all; a VM autoscaling group can set a minimum of zero but pays a full boot on the way back, which is the slowest form of this trade.
- GENERALThe underlying exchange — idle cost against first-request latency — is universal, including for self-hosted services stopped overnight and for agent workloads whose model backends must be loaded before use.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — deciding which paths are synchronous, which is what determines whether a start-up pause is acceptable anywhere.