Scale This System

A web application on one 4-CPU instance and a Postgres on another. It serves the first few thousand users comfortably. Four metrics tell you when it stops: application CPU and p99 latency, database CPU and read/write ratio, connection count, and queue depth for anything asynchronous — which, right now, does not exist. Nothing here should change until one of these moves. A missing index (Why Is This Query Slow? Indexes) is checked before every rung below, because it is a 100× win that adds no component at all.

Symptom: app CPU at 90% during the daily peak, p99 from 200 ms to 1.4 s, one deploy per day means a minute of 502s. Fix: first, resize (vertical, see Horizontal vs Vertical Scaling); when the next resize doubles the bill or the single machine is an unacceptable failure domain, put a Load Balancing tier in front and run three instances. Sessions in process memory break immediately, so they move to Redis — the Stateless vs Stateful Services change comes with this rung, not later.

New problem: three instances each open a connection pool, and the database has 3× the connections and 3× the read traffic; nothing about the database got faster. Also new: a balancer to keep highly available, health checks to get right, and a Redis that is now tier-one because it holds every session.

Symptom: database CPU at 80%, 95% of queries are reads, the top ten queries by frequency are the same product and profile lookups, pg_stat_activity shows connections waiting. Fix, cheapest first: a connection pooler (PgBouncer) so 3 × 50 app connections become 20 real ones; then a cache in front of the hot reads (Caching Architecture, patterns in Caching Patterns) — the same Redis, now holding product:{id} with a 60 s jittered TTL, which removes perhaps 90% of those reads; then a read replica (Replication and Read Scaling) for the reads that remain and for reporting queries that should never touch the primary.

New problem: two of them. The cache introduces staleness and the invalidation question — who deletes product:42 when the price changes (Cache Invalidation, Stampedes and Hot Keys)? — and it is now load-bearing: if Redis dies, the database receives 10× its normal reads. The replica introduces replication lag: a user saves their profile, the next request reads a replica 300 ms behind, and they see the old value. Route reads-after-writes to the primary, or wait for the replica’s LSN.

Symptom: the primary is at 70% CPU on writes alone; replicas do not help because every write is replayed on every replica; the events table is 2 TB and VACUUM no longer finishes overnight. Fix: before sharding, three cheaper moves — scale the primary vertically (more RAM, NVMe), partition the huge table by time so old partitions can be detached and indexes stay small, and move write-heavy, low-value data (analytics events, logs) out of Postgres entirely into a queue and a column store. Only when a single primary cannot absorb the core write rate does sharding (Partitioning and Sharding) enter: split users across N databases by a shard key — hashed with Consistent Hashing so adding a shard moves 1/N of the keys — and accept what that costs.

New problem: sharding is the most expensive rung on the ladder. Cross-shard joins become application-side merges, a transaction touching two users’ shards is a distributed transaction, global uniqueness needs a coordinator or a key scheme, and rebalancing is a project. This is why the capstone treats sharding as a *consideration*: the honest design records the shard key you would use and the write rate at which you would use it, and does not shard before that rate is measured.

Symptom: page load in Singapore is 2.8 s against 600 ms in the origin region; 70% of origin bandwidth is images, JS and fonts that never change; a product launch tripled bandwidth and the app instances spent their CPU serving files. Fix: hashed asset URLs with Cache-Control: immutable, uploads served from object storage, and a CDN Architecture in front of both, with origin shielding so a launch produces one origin fetch per asset instead of one per edge. Anonymous product pages get max-age=60, stale-while-revalidate, so the CDN absorbs their reads too.

New problem: cache keys and purge. A per-user page accidentally marked public leaks accounts; a deploy that switches HTML before assets are uploaded serves 404s; a purge-everything on deploy sends the whole miss storm to the origin. The CDN is also a second place where staleness lives, so a product price change now has to reach Redis *and* the edge.

Symptom: POST /orders p99 is 3.2 s because the handler sends a confirmation email, renders a PDF invoice, calls a fraud-scoring API and updates the analytics warehouse before responding; a 30 s email-provider outage becomes a checkout outage. Fix: the request writes the order and enqueues the consequences (Message Queues, Background Jobs and Workers); a worker pool consumes them, retries with backoff, and dead-letters what fails five times. The response returns in 80 ms. Workers scale horizontally on queue depth, independently of the web tier.

New problem: asynchrony. The user sees "order placed" before the email exists; the job runs at least once, so a worker that crashes after sending must not send again on retry — the handler is made idempotent with a processed-job table or an idempotency key. The queue can also hide a capacity shortfall: if producers outrun workers, the oldest message ages into hours before anyone notices; queue age, not queue length, is the alert (Backpressure).

Read the table bottom-up as a diagnosis tool: given a symptom, which rung is next? Read it top-down as a review tool: for each box in a proposed design, which symptom justified it? A box with no symptom is a cost with no benefit. The same discipline, applied to the database alone, is Scaling from One User to Millions.