Horizontal vs Vertical Scaling
A bigger machine is simpler and has a ceiling; more machines have no ceiling and require statelessness, a load balancer and coordination you did not have before.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The service is at capacity. Do I make the machine bigger or add more of them?
Traffic has doubled and latency is rising. We need more capacity by the end of the week, and a plan for the next doubling.
Add instances. Horizontal scaling is what real systems do, and vertical scaling is a stopgap that does not scale.
The service has instance-local state, so adding instances changes behaviour rather than adding capacity (Making an Existing Service Stateless).
- The service has instance-local state, so adding instances changes behaviour rather than adding capacity (Making an Existing Service Stateless).
- Each new instance opens its own connection pool, so tripling instances triples connections and the database hits its limit before the fleet hits its capacity (Connection Pools).
- The bottleneck was never the application: the database was already saturated, so more application instances add load to the thing that was slow and make it worse (Cascading Failure).
- In-process caches now have a third of the hit rate each, so the same traffic produces three times the cache misses and more load downstream (Local vs Distributed Cache).
- A simple, urgent capacity problem becomes a multi-week statelessness project, when doubling the instance size would have bought a month.
What is actually happening
- Vertical scaling gives one process more of a resource: cores, memory, I/O bandwidth. Nothing about the application changes, which is exactly why it is fast to do and why it is often the correct first move.
- Its ceiling is real but distant, and it arrives in two forms: the largest machine available, and the point where the application stops using extra cores — a single-threaded event loop, a global interpreter lock, or a lock that serialises the hot path (Backend Runtime Models).
- A vertical resize usually requires a restart, which means a brief interruption unless there is already more than one instance — and one instance is a single point of failure regardless of its size.
- Horizontal scaling adds processes. Capacity becomes a number, failure of one instance is survivable, and rolling deploys become possible. The prerequisite is that any instance can serve any request (Stateless Services).
- Its cost is coordination: a load balancer with health checks, shared state for anything that used to be local, N times the connections and outbound calls to every dependency, and a deploy that runs two versions at once (Rolling Deployments).
- Neither scales the database. Application scaling almost always moves the bottleneck rather than removing it, and the next bottleneck is usually the data layer (Read Replicas From the Application).
- The two compose: most production services are horizontally scaled at a vertically-chosen instance size, and both numbers are tunable.
Two shapes of the same graph
Both approaches add capacity; they differ in what else they add. Vertical adds nothing else — same process, same assumptions, same deploy. Horizontal adds a coordination layer, and every subsequent problem in this module exists because of it: load balancing, sticky sessions, autoscaling signals, replication lag.
The comparison worth internalising is not "which is better" but "what does each one make me responsible for next".
Choosing honestly
The decision is usually settled by two facts that are cheap to establish: which resource is actually saturated, and whether the service is stateless yet. With both answers, the choice is nearly mechanical.
Note the last two options. "Neither" is a real answer more often than the framing admits — a missing index or an N+1 query is a capacity problem solved by removing work rather than by adding machines.
What is saturated, and what does the service already support?
when Service is stateful, the need is urgent, and the instance is not near the largest available.
cost A restart, a ceiling later, and a single point of failure if you stay at one instance.
when Service is stateless, dependencies have headroom, and availability matters.
cost Load balancer, health checks, N times connections and outbound calls, two-version deploys.
when Steady growth with a known traffic profile.
cost Two knobs to tune and to re-tune as the workload changes.
when Single-threaded or GIL-bound runtime with spare cores.
cost Per-worker memory and per-worker connections; local state now breaks at worker granularity (Worker Processes).
when The database is the saturated component.
cost Replicas, caching or partitioning — a different project with different risks (Read Replicas From the Application).
when An N+1, a missing index, an unbounded query or a synchronous call that could be deferred.
cost Requires a diagnosis, and is usually the cheapest capacity available (The N+1 Query Problem).
The arithmetic that turns a scale-out into an incident
The most common self-inflicted injury when scaling out is forgetting that every instance carries its own copy of everything the process held. Connection pools are the sharpest case, because the database enforces a hard limit and refuses connections rather than queueing them.
The calculation must include the deploy window, when old and new instances coexist, and the autoscaler's maximum rather than its current count. Both are routinely omitted, and both are the moment the limit is actually reached.
// Config copied unchanged as the fleet grew:
const pool = new Pool({ max: 20 })
// 3 instances -> 60 connections (fine)
// autoscaler max 12 instances -> 240
// during a rolling deploy, up to 14 -> 280
// plus workers (4 x 10) -> 40
// plus migrations and admin -> ~5
//
// Postgres max_connections = 200.
// The service scales up and the database
// starts refusing connections — which reads
// as "the database is down".// Budget backwards from the database limit.
//
// max_connections 200
// - superuser reserved -3
// - admin/migrations -5
// - workers (4 x 10) -40
// = available to the API 152
//
// max instances during deploy: 14
// 152 / 14 = 10 per instance
const pool = new Pool({ max: 10 })
// Above ~15 instances, per-instance pools stop
// working at all: put a pooler in front and let
// it multiplex ([[connection-pools]]).The database enforces a connection limit and refuses beyond it — there is no graceful degradation, only failure. Pool size is therefore a property of the *fleet*, not of the instance, and the fleet's size is its maximum during a deploy, not its current count. Past a certain instance count no per-instance division works, and multiplexing has to move outside the application.
How to build it
Most important first.
- Find the bottleneck before adding anything. Adding capacity to a component that is not saturated changes nothing except cost (Why Is My API Slow?).
- If the service is not yet stateless, scale vertically now and do the statelessness work deliberately. Emergency horizontal scaling of a stateful service produces intermittent correctness bugs on top of a capacity problem.
- When scaling horizontally, do the connection arithmetic first: instances times pool size must fit inside the database connection budget, with headroom for a rollout when both old and new instances exist (Connection Pools).
- Prefer more, smaller instances over fewer, larger ones when the runtime cannot use many cores in one process; prefer fewer, larger when per-instance overhead (caches, pools, warm state) is significant.
- Keep at least two instances regardless of capacity. The second one is for availability and for rolling deploys, not for throughput.
- Right-size the instance to the runtime's actual concurrency model. Giving 16 cores to a single-threaded process buys nothing; running 16 worker processes on it might (Worker Processes).
- Re-measure after each change. Scaling moves the bottleneck, and the new bottleneck is rarely where the last one was (The Bottleneck Moves After Every Fix in Observability & Performance).
What can go wrong
- Scaling out a service whose dependency is the bottleneck, converting a slow service into an outage for everything that shares that dependency.
- Connection exhaustion caused by the scale-out itself, which looks like a database failure and is a pool-arithmetic error (Connection Pool Exhaustion).
- Cache hit rate collapse as a local cache is divided across more instances, increasing downstream load exactly when capacity was scarce.
- A vertical resize that requires downtime, discovered during the change window because there was only one instance.
- Scaling to a size where per-instance startup (image pull, warmup, pool construction) is slower than the traffic ramp, so capacity always arrives late (Autoscaling a Backend).
- Assuming linear returns: doubling instances rarely doubles throughput, because shared dependencies, locks and coordination do not scale with the fleet (Why Eight Cores Give You Four and a Half in Concurrency & Parallelism).
- Adding instances multiplies the number of concurrent writers to the same rows, so latent races that were rare at one instance become routine (Backend Races).
- A scale-out and a rolling deploy overlapping means new instances start on one version while the fleet is mid-migration to another (Rolling Deployments).
- More instances means more identities and more credentials in circulation. Prefer platform-issued short-lived workload identity over static keys copied into every instance (Secrets Are Not Configuration).
- A larger fleet enlarges the attack surface only if each instance is independently exposed; keep instances private behind the load balancer rather than individually reachable (Public Exposure, Read With Context in Cloud & Infrastructure).
- A single large instance concentrates blast radius: compromise gives access to everything it held, including a larger in-memory working set.
- "Vertical scaling does not scale." It scales a long way, immediately, with no code change. It has a ceiling; most services never approach it.
- "Horizontal scaling is linear." It is sublinear in practice: shared dependencies, lock contention and coordination all take a share, and the shape of the curve is the interesting engineering (Why Eight Cores Give You Four and a Half in Concurrency & Parallelism).
- "We scaled the service, so it is faster." Scaling adds capacity. It reduces queueing latency, and it does not make any single request execute faster (Little's Law as Working Intuition in Observability & Performance).
- "Add instances until it is fast." If the bottleneck is downstream, adding instances makes it slower — and the graph will show that only after the incident.
Operating it
- Measure saturation of the actual constrained resource — CPU, memory, pool waiters, event-loop lag, thread-pool queue — not just utilisation (USE: Utilization, Saturation, Errors in Observability & Performance).
- Plot throughput against instance count. If the line flattens, the constraint is shared and adding instances is no longer buying capacity.
- Watch total connections to each dependency as a function of instance count. This is the number that turns a scale-out into an incident.
- Track per-instance cache hit rate before and after a scale-out; a drop explains a latency increase that instance metrics alone will not.
- At 10x, horizontal scaling is routine if the service is stateless and the dependencies were sized for the fleet, not for one instance.
- At 100x, coordination costs become the subject: connection multiplexing, request coalescing, partitioning the data layer, and reducing per-instance overhead (Request Coalescing).
- At every scale, the honest question is which single resource is saturated. "Scaling" without that answer is an expenditure, not a fix.
- Vertical: no application change, no coordination, immediate — and a ceiling, a restart to resize, and a single point of failure if instance count stays at one.
- Horizontal: no ceiling, survives instance failure, enables rolling deploys — and requires statelessness, a load balancer, N times the dependency pressure and a two-version deploy story.
- Doing both means tuning two numbers instead of one, which is more knobs and better fit.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe trade — simplicity and a ceiling against coordination and no ceiling — holds for any service.
- RUNTIME-SPECIFICHow much a bigger machine buys depends on the concurrency model: a single-threaded event loop gains little from extra cores without multiple worker processes, a GIL-bound Python process gains nothing on CPU work without processes, and a Go or JVM service uses additional cores directly (Backend Runtime Models).
- SCALE-SPECIFICBelow a few instances, the availability argument dominates the capacity argument: run two even if one has enough capacity, because one is a single point of failure and blocks rolling deploys.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.