From a Simple App to a Scaled System
A system grows from browser → backend → database into gateways, services and caches one measured problem at a time — CPU at 90%, p99 above 800 ms, four teams blocked on one deploy — and every step buys capacity by adding a problem you did not have before.
Teams either under-build (one box until it falls over at 2 a.m.) or over-build (twelve services for a thousand users). Treating architecture as a sequence of responses to measured problems gives a way to know which step is next and when it is actually due.
Stage 1: browser → backend → database
The starting picture is three boxes: a browser, one backend process, one database. It is not a toy. A single 4-vCPU instance running a reasonably written Node, Go or Python service handles on the order of 1,000–3,000 simple requests per second, and a single Postgres on a machine with the working set in RAM handles several thousand transactions per second. For most products that is years of headroom. The architecture has one failure domain (the box), one deployment (restart the process), one transaction scope (the database), and no distributed-systems problems at all.
The right move at this stage is to instrument it, not to grow it: request rate, CPU, p50/p99 latency per endpoint, database CPU and slowest queries. Those numbers are what will tell you which step is next. Without them, every later decision is a guess dressed up as a plan.
Stage 2: the first measured problems
The first wall is usually backend CPU: the graph shows 90% sustained during the evening peak, and p99 latency climbs from 120 ms to 800 ms because requests queue for a core. The fix is a load balancer in front of N backend instances. That step is only possible if the backend is stateless — if sessions live in process memory, users are logged out at random as the balancer sends them to a different instance (Stateless vs Stateful Services; the challenge logged-out-after-scaling is this exact incident). So the honest cost of the step is: move sessions to Redis, and now you own a Redis.
The second wall is the database. Query logs show the same product catalogue and user profile reads at 90% of all queries, and database CPU is at 80% while writes are a small fraction. A cache in front of the database (Redis, with a TTL of, say, 60 s for the catalogue) removes most of that read load. The new problem is staleness: a price change is invisible for up to 60 s unless you invalidate on write — see Caching Architecture and Cache Invalidation, Stampedes and Hot Keys. If reads are still too heavy, read replicas (Replication and Read Scaling) add capacity and hand you replication lag.
Stage 3: the organisational problem
The third wall is often not technical. The company now has four teams — accounts, orders, payments, growth — in one codebase. The build takes 25 minutes, the test suite 40, and a payments change needs a full regression because nobody is sure what it touches. Deploys happen twice a week because merging is where the four teams collide; a bug in a growth experiment rolls back the payments fix shipped in the same release. The measured problem is deploy conflicts and lead time, not CPU.
This is the problem that justifies splitting the application: clients → API gateway → User / Order / Payment services, each with its own database and its own deploy pipeline (Microservices, API Gateway). The gateway handles authentication, rate limiting and routing at the edge. Each team ships on its own schedule. The cost is everything the next lessons are about: every in-process call between orders and payments is now a network call with a timeout; an order and its payment can no longer be committed in one transaction (Distributed Transactions); a request that was one log line is now three services and a trace (Distributed Tracing). The split is worth it only when the organisational pain is measurably larger than that.
Notice the alternative that is often better: a Modular Monolith gives the four teams enforced boundaries inside one deployable, fixing most of the collisions without a single network hop. Many systems should stop there.
The rule: every box has a cause, and every cause has a number
Read the three stages as a table and the pattern is explicit. Each row is a symptom you can see on a dashboard, the step that answers it, and the new problem that step introduces. The step is justified only when the symptom is measured, and the new problem must be owned the day the step ships — a cache without an invalidation plan or a service split without tracing is a step half taken. The capstone Scale This System walks this ladder in detail, including the rungs (CDN, queues and workers, partitioning) that this lesson skips.
Two things this framing prevents. First, premature steps: nobody adds a gateway and three services to a system whose backend CPU is at 12%. Second, wrong steps: a slow database whose real problem is one missing index (Why Is This Query Slow? Indexes) is fixed with an index, not with read replicas — the ladder in Scaling from One User to Millions starts with indexes for that reason.
| Measured symptom | Step | What it buys | New problem you now own |
|---|---|---|---|
| Backend CPU 90% at peak; p99 120 ms → 800 ms | Load balancer + N instances | Linear request capacity | Sessions must leave the process; the balancer is a new component |
| Same 20 queries are 90% of DB load; DB CPU 80% | Cache (Redis, TTL) | Most reads never reach Postgres | Staleness and invalidation; a stampede when a hot key expires |
| Reads still saturate the primary | Read replicas | Near-linear read capacity | Replication lag: users reading their own stale writes |
| Build 25 min, test 40 min, 4 teams block each other, deploys twice a week | Modular monolith, then extract services | Independent deploys per team | Network calls, partial failure, no cross-service transactions, tracing |
| Email/report work makes checkout p99 spike | Queue + workers | Request path no longer waits on slow work | At-least-once delivery: handlers must be idempotent |
Key points
- Browser → backend → database is a real architecture with years of headroom for most products; instrument it before growing it.
- Each step is triggered by a measured symptom: CPU at 90% → load balancer; read-heavy DB → cache/replicas; team collisions → boundaries.
- Every rung buys capacity by adding a problem — sessions leave the process, caches go stale, replicas lag, services lose transactions.
- The split into services is usually justified by an organisational problem (deploy conflicts, lead time), not by traffic.
- A modular monolith fixes most team collisions without network hops; many systems should stop there.
Grow a system one problem at a time
Architecture is not chosen up front; it is the residue of problems you actually had. The next five stages each start from a number on a dashboard.
How data moves through it
One request or event, hop by hop.
- 1Client → API gateway: TLS terminated, JWT verified, rate limit checked, request routed by path prefix.
- 2Gateway → Order service:
POST /orders; the order service validates and writes apendingorder to the Order DB. - 3Order service → Payment service: synchronous
chargecall with a 3 s timeout and an idempotency key derived from the order id. - 4Payment service → Payment provider: HTTPS call; the result is written to the Payment DB before the response is returned.
- 5Order service → Order DB: mark the order
paid, and publishOrderPaidfor downstream consumers (email, analytics) asynchronously.
When to use — and when not
- When a dashboard shows a specific saturated resource — CPU, database load, deploy lead time — and you need to choose the next step rather than the final picture.
- When explaining or defending a design: present it as the sequence of problems that produced it.
- When a team proposes a target architecture: ask which stage the current metrics place you at.
- Do not evolve on projected traffic; build for roughly 10× current load, not 1,000×, because each step costs a new class of bug.
- Do not skip rungs: a service split before indexes, a pool and a cache have been tried adds the hardest problem first.
- Do not apply the ladder to a system with no metrics — instrument first, or every step is a guess.
Tradeoffs
Averages across the three stages. Stage 1 scores 1 on complexity and 5 on consistency; stage 3 scores 4–5 on complexity and operational cost and drops consistency to eventual across services. The evolution is a slide along those axes, taken one notch at a time.
How it fails
- Scaling out a stateful backend: round-robin sends a user to an instance without their session, and 1 in N requests is unauthenticated.
- Caching without invalidation: a price change is invisible for the TTL, and a hot key expiring under load triggers thousands of identical recomputes (
midnight-cache-stampede). - Splitting services before boundaries exist: the order and payment services share a database and deploy together — a distributed monolith with all the cost and none of the independence.
- Adding a rung to fix the wrong symptom: replicas for a database whose real problem is one missing composite index.
How it scales
- Stateless backends scale linearly behind a load balancer until the shared database becomes the limit.
- The database scales first vertically, then by read replicas and caching for reads, then by partitioning and sharding for writes — a separate ladder (Scaling from One User to Millions).
- Services scale independently once split: the payment service can run 2 instances while the catalogue runs 20 — the one technical argument for the split that is not organisational.
- The gateway and load balancer become the new single points of failure and are made redundant (pairs, DNS, anycast) — see Load Balancing.
How it interacts with databases, queues, caches, APIs and external systems
- Database: one Postgres at stage 1; a primary plus replicas at stage 2; a database per service at stage 3, with no cross-database joins or transactions.
- Cache: Redis holds sessions (so instances are interchangeable) and hot reads (so the database is not the ceiling); both must be survivable if Redis is lost.
- Queue: appears when slow work (email, reports) is measured on the request path; it moves that work off the path at the price of at-least-once delivery.
- API gateway: the single entry point at stage 3, owning authentication, rate limiting and routing — and nothing that belongs to a service.
- External systems: the payment provider is reached only from the payment service, with a timeout, a retry budget and a webhook for asynchronous confirmations.