Fundamentalsevolutionscalingload balancercacheapi gateway

From a Simple App to a Scaled System

A system grows from browser → backend → database into gateways, services and caches one measured problem at a time — CPU at 90%, p99 above 800 ms, four teams blocked on one deploy — and every step buys capacity by adding a problem you did not have before.

▶ InteractiveInterview question
Progress
What problem does this solve?

Teams either under-build (one box until it falls over at 2 a.m.) or over-build (twelve services for a thousand users). Treating architecture as a sequence of responses to measured problems gives a way to know which step is next and when it is actually due.

Stage 1: browser → backend → database

The starting picture is three boxes: a browser, one backend process, one database. It is not a toy. A single 4-vCPU instance running a reasonably written Node, Go or Python service handles on the order of 1,000–3,000 simple requests per second, and a single Postgres on a machine with the working set in RAM handles several thousand transactions per second. For most products that is years of headroom. The architecture has one failure domain (the box), one deployment (restart the process), one transaction scope (the database), and no distributed-systems problems at all.

The right move at this stage is to instrument it, not to grow it: request rate, CPU, p50/p99 latency per endpoint, database CPU and slowest queries. Those numbers are what will tell you which step is next. Without them, every later decision is a guess dressed up as a plan.

Stage 1: one process, one database
HTTPSSQLBrowserBackendPostgres
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

Stage 2: the first measured problems

The first wall is usually backend CPU: the graph shows 90% sustained during the evening peak, and p99 latency climbs from 120 ms to 800 ms because requests queue for a core. The fix is a load balancer in front of N backend instances. That step is only possible if the backend is stateless — if sessions live in process memory, users are logged out at random as the balancer sends them to a different instance (Stateless vs Stateful Services; the challenge logged-out-after-scaling is this exact incident). So the honest cost of the step is: move sessions to Redis, and now you own a Redis.

The second wall is the database. Query logs show the same product catalogue and user profile reads at 90% of all queries, and database CPU is at 80% while writes are a small fraction. A cache in front of the database (Redis, with a TTL of, say, 60 s for the catalogue) removes most of that read load. The new problem is staleness: a price change is invisible for up to 60 s unless you invalidate on write — see Caching Architecture and Cache Invalidation, Stampedes and Hot Keys. If reads are still too heavy, read replicas (Replication and Read Scaling) add capacity and hand you replication lag.

Stage 2: copies behind a balancer, cache in front of the database
hot readswritesreadsWAL streamBrowserLoad balancerBackend 1Backend 2Backend 3Redis (sessions + cache)Postgres primaryRead replica
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

Stage 3: the organisational problem

The third wall is often not technical. The company now has four teams — accounts, orders, payments, growth — in one codebase. The build takes 25 minutes, the test suite 40, and a payments change needs a full regression because nobody is sure what it touches. Deploys happen twice a week because merging is where the four teams collide; a bug in a growth experiment rolls back the payments fix shipped in the same release. The measured problem is deploy conflicts and lead time, not CPU.

This is the problem that justifies splitting the application: clients → API gatewayUser / Order / Payment services, each with its own database and its own deploy pipeline (Microservices, API Gateway). The gateway handles authentication, rate limiting and routing at the edge. Each team ships on its own schedule. The cost is everything the next lessons are about: every in-process call between orders and payments is now a network call with a timeout; an order and its payment can no longer be committed in one transaction (Distributed Transactions); a request that was one log line is now three services and a trace (Distributed Tracing). The split is worth it only when the organisational pain is measurably larger than that.

Notice the alternative that is often better: a Modular Monolith gives the four teams enforced boundaries inside one deployable, fixing most of the collisions without a single network hop. Many systems should stop there.

Stage 3: gateway and services, each owning its data
auth, rate limitcharge (network, timeout)HTTPSWeb + mobile clientsAPI gatewayUser serviceOrder serviceUser DBPayment serviceOrder DBPayment DBPayment provider
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

The rule: every box has a cause, and every cause has a number

Read the three stages as a table and the pattern is explicit. Each row is a symptom you can see on a dashboard, the step that answers it, and the new problem that step introduces. The step is justified only when the symptom is measured, and the new problem must be owned the day the step ships — a cache without an invalidation plan or a service split without tracing is a step half taken. The capstone Scale This System walks this ladder in detail, including the rungs (CDN, queues and workers, partitioning) that this lesson skips.

Two things this framing prevents. First, premature steps: nobody adds a gateway and three services to a system whose backend CPU is at 12%. Second, wrong steps: a slow database whose real problem is one missing index (Why Is This Query Slow? Indexes) is fixed with an index, not with read replicas — the ladder in Scaling from One User to Millions starts with indexes for that reason.

Symptom → step → new problem
Measured symptomStepWhat it buysNew problem you now own
Backend CPU 90% at peak; p99 120 ms → 800 msLoad balancer + N instancesLinear request capacitySessions must leave the process; the balancer is a new component
Same 20 queries are 90% of DB load; DB CPU 80%Cache (Redis, TTL)Most reads never reach PostgresStaleness and invalidation; a stampede when a hot key expires
Reads still saturate the primaryRead replicasNear-linear read capacityReplication lag: users reading their own stale writes
Build 25 min, test 40 min, 4 teams block each other, deploys twice a weekModular monolith, then extract servicesIndependent deploys per teamNetwork calls, partial failure, no cross-service transactions, tracing
Email/report work makes checkout p99 spikeQueue + workersRequest path no longer waits on slow workAt-least-once delivery: handlers must be idempotent

Key points

  • Browser → backend → database is a real architecture with years of headroom for most products; instrument it before growing it.
  • Each step is triggered by a measured symptom: CPU at 90% → load balancer; read-heavy DB → cache/replicas; team collisions → boundaries.
  • Every rung buys capacity by adding a problem — sessions leave the process, caches go stale, replicas lag, services lose transactions.
  • The split into services is usually justified by an organisational problem (deploy conflicts, lead time), not by traffic.
  • A modular monolith fixes most team collisions without network hops; many systems should stop there.

Grow a system one problem at a time

Grow a system one problem at a time
Six stages of one shop. Each box on the diagram was added to fix a measured symptom — and each fix introduced the next problem.
Stage 1
BrowserBackendPostgreSQL
Traffic
50 rps
p99 latency
120 ms
DB CPU
8%
Deploys / day
5
Teams
1
Components
3
A browser, a backend, a database
Symptom. No symptom. 50 requests per second, p99 120 ms, one deploy a few times a day. It works.
Fix. Nothing. Resist the urge to add anything: every box you add later is paid for by a problem you can measure.
New problem this step introduced. One process is a single point of failure and a deploy is a few seconds of downtime. At 50 RPS that is a tolerable trade, not a bug.

Architecture is not chosen up front; it is the residue of problems you actually had. The next five stages each start from a number on a dashboard.

1/6 · 1. A browser, a backend, a database

How data moves through it

One request or event, hop by hop.

  1. 1Client → API gateway: TLS terminated, JWT verified, rate limit checked, request routed by path prefix.
  2. 2Gateway → Order service: POST /orders; the order service validates and writes a pending order to the Order DB.
  3. 3Order service → Payment service: synchronous charge call with a 3 s timeout and an idempotency key derived from the order id.
  4. 4Payment service → Payment provider: HTTPS call; the result is written to the Payment DB before the response is returned.
  5. 5Order service → Order DB: mark the order paid, and publish OrderPaid for downstream consumers (email, analytics) asynchronously.

When to use — and when not

Use it when
  • When a dashboard shows a specific saturated resource — CPU, database load, deploy lead time — and you need to choose the next step rather than the final picture.
  • When explaining or defending a design: present it as the sequence of problems that produced it.
  • When a team proposes a target architecture: ask which stage the current metrics place you at.
Avoid it when
  • Do not evolve on projected traffic; build for roughly 10× current load, not 1,000×, because each step costs a new class of bug.
  • Do not skip rungs: a service split before indexes, a pool and a cache have been tried adds the hardest problem first.
  • Do not apply the ladder to a system with no metrics — instrument first, or every step is a guess.

Tradeoffs

Complexity
low → high
Ops cost
low → high
Latency
low → high
Consistency
weak → strong
Scalability
poor → strong

Averages across the three stages. Stage 1 scores 1 on complexity and 5 on consistency; stage 3 scores 4–5 on complexity and operational cost and drops consistency to eventual across services. The evolution is a slide along those axes, taken one notch at a time.

How it fails

  • Scaling out a stateful backend: round-robin sends a user to an instance without their session, and 1 in N requests is unauthenticated.
  • Caching without invalidation: a price change is invisible for the TTL, and a hot key expiring under load triggers thousands of identical recomputes (midnight-cache-stampede).
  • Splitting services before boundaries exist: the order and payment services share a database and deploy together — a distributed monolith with all the cost and none of the independence.
  • Adding a rung to fix the wrong symptom: replicas for a database whose real problem is one missing composite index.

How it scales

  • Stateless backends scale linearly behind a load balancer until the shared database becomes the limit.
  • The database scales first vertically, then by read replicas and caching for reads, then by partitioning and sharding for writes — a separate ladder (Scaling from One User to Millions).
  • Services scale independently once split: the payment service can run 2 instances while the catalogue runs 20 — the one technical argument for the split that is not organisational.
  • The gateway and load balancer become the new single points of failure and are made redundant (pairs, DNS, anycast) — see Load Balancing.

How it interacts with databases, queues, caches, APIs and external systems

  • Database: one Postgres at stage 1; a primary plus replicas at stage 2; a database per service at stage 3, with no cross-database joins or transactions.
  • Cache: Redis holds sessions (so instances are interchangeable) and hot reads (so the database is not the ceiling); both must be survivable if Redis is lost.
  • Queue: appears when slow work (email, reports) is measured on the request path; it moves that work off the path at the price of at-least-once delivery.
  • API gateway: the single entry point at stage 3, owning authentication, rate limiting and routing — and nothing that belongs to a service.
  • External systems: the payment provider is reached only from the payment service, with a timeout, a retry budget and a webhook for asynchronous confirmations.