The question this answers
What is the actual sequence of problems that turns an application on a laptop into a production system, and what does each one force you to add?
The same API and relational database from What Infrastructure Actually Is, now with real users: first ten, then a thousand, then a hundred thousand across two continents — with the team growing from one engineer to fifteen along the way.
A defensible order of adoption. At every stage the system is coherent, operable and complete for the load it serves, and the next stage is triggered by an observed failure rather than by ambition.
Seven stages, each forced by the failure of the last
The progression below is not a maturity ladder to climb. It is a list of problems, in the order they usually arrive, with the smallest fix for each. Most systems stop somewhere in the middle and are correct to stop there — a business with two thousand customers in one country has no problem that stage six solves, and adopting it anyway means paying for a solution and inheriting its failure modes for free.
The discipline is the forced by column. If you cannot fill it in from something that actually happened — a deploy that dropped requests, a disk that filled, a night nobody noticed the process had died — then the stage has not arrived yet. "We will need it eventually" is not a forcing event; it is a plan to be surprised later about which thing you actually needed.
Two things are worth noticing about the shape of this list. First, the early stages fix *availability of one machine* and the later ones fix *availability of one location* — those are different problems and they cost different amounts. Second, every stage adds an operational obligation that never goes away. You do not get to add a load balancer and then stop thinking about it; you have added a certificate that expires, a health check that can be wrong, and a component that can be misconfigured to route to nothing.
- 10 · Laptopday 0
App and database on one machine. Restarted by hand, reachable only by you, backed up by whatever backs up your laptop.
Nothing outside the room can reach it, and "it works on my machine" is literally the only thing that is true.
- 21 · One remote serverweeks 1–10
A single rented VM. App and database side by side, a process supervisor, a DNS A record, a TLS certificate, a nightly database dump copied off the box.
Forced by: nobody else could use it. Now: one machine is the whole system. A reboot is an outage; a full disk takes the database with it; a lost machine loses everything since last night's dump.
- 32 · Split the database outweeks 10–16
Database moved to a managed instance with automated backups and point-in-time recovery. The app server keeps only replaceable state.
Forced by: an application memory leak took the database process down with it, and the last dump was 19 hours old. Now: the app instance is disposable but still singular.
- 43 · Load balancer and two instancesmonths 4–6
A load balancer with a stable address and TLS termination, two identical app instances in two zones, health checks, graceful shutdown on deploy.
Forced by: every deploy dropped in-flight requests, and one instance restart was a visible outage. Now: sessions and uploaded files stored on local disk break, because requests land on either instance.
- 54 · Object storage, cache, background workersmonths 6–12
Uploads to object storage with signed URLs; a cache in front of an expensive read path; a queue and worker pool for anything longer than a request.
Forced by: files vanished when an instance was replaced; the report endpoint held connections for 90 s and starved the pool. Now: three more components to secure, monitor and pay for.
- 65 · IaC, CI/CD, real observabilitymonths 9–18
Infrastructure defined in code and reviewed; a pipeline that builds one artifact and promotes it; metrics, logs, traces and alerts that reach a human.
Forced by: a hand-made staging environment diverged from production and a deploy failed in a way nobody could reproduce. Now: the pipeline itself is production infrastructure with its own identity and blast radius.
- 76 · Autoscaling, CDN, multi-zone by defaultyear 2
Metric-driven instance scaling with warm headroom, a CDN for static and cacheable responses, every tier spread across zones, database with a standby in a second zone.
Forced by: Monday-morning peaks at 20x the mean, and a zone incident that removed half of a two-instance fleet. Now: scaling lag becomes the failure mode — new capacity is minutes late.
- 87 · Multi-regionyear 3+, or never
A second region serving a second continent, with a deliberate answer for where writes go and what is lost on failover.
Forced by: a stated latency requirement for users 8,000 km away, or a regulatory requirement, or an RTO that a single region cannot meet. Now: you own a distributed data problem, permanently. Most systems should never reach this stage — see Multi-Region Deployment.
Stage 3 in detail: what a load balancer actually costs you
Stage 3 is worth expanding because it is where most teams first meet the general shape of the trade. The forcing problem is precise: with one instance, systemctl restart is a 4-second outage and a bad deploy is a total one. Two instances behind one address fixes it — and immediately breaks three things that worked fine before.
Sessions stored in process memory now fail half the time, because the second request lands on the other instance. Files written to local disk are visible to one instance only. Any scheduled job compiled into the application now runs twice, which is harmless for a cache warm and catastrophic for a billing run. None of these are load-balancer bugs; they are assumptions that were true when the fleet size was one, and the fleet size is a load-bearing assumption nobody wrote down.
The fix set is well known — externalize session state, move files to object storage, move scheduled work to a single leader or a queue — and each is a small piece of work if you know it is coming and an incident if you do not. This is the general shape: *the component solves the stated problem and invalidates an unstated assumption.* Ask, for every stage, what the previous stage let you assume.
- Sessions in process memory break: the second request lands elsewhere. Externalize them or use signed stateless tokens.
- Local-disk writes become per-instance and are lost on replacement. This is what forces object storage, not "best practice".
- In-process cron now runs on every instance. Harmless for cache warming, a duplicated invoice run for anything that charges money.
- A certificate now exists and expires. Automate renewal at stage 3, not after the first expiry incident.
- Health checks become load-bearing: a check that only proves the process is alive will happily route traffic into an instance that cannot reach the database. See Liveness vs Readiness.
The cost curve is not smooth
Each stage changes the cost *shape*, not just the amount. Stage 1 is one predictable fixed line. Stage 3 adds a load balancer that bills hourly plus per processed gigabyte, and doubles instance-hours to buy redundancy — you are paying roughly twice as much to serve the same traffic, and that is correct, because you are buying availability rather than capacity. Stage 6 introduces a genuinely non-linear line: egress and CDN, which scale with users rather than with instances.
The item that surprises teams at stage 7 is cross-region data transfer. Replicating writes between two regions bills per gigabyte, continuously, in both directions if the design is active-active — and it is charged at the most expensive transfer rate in most providers' price lists. A team that adopts multi-region for latency and then discovers it has doubled its data-layer bill and acquired a replication-lag failure mode has usually not made the trade it thought it was making.
The useful discipline: at every stage, name the new *meter*, not the new amount. Meters are what you inherit permanently; amounts change with traffic.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Every stage exists because the previous stage failed in a specific, observable way. Without that event, the stage has not arrived.
- Stopping in the middle is a valid and common outcome; most systems have no problem that multi-region solves.
- Each component solves the stated problem and invalidates an unstated assumption — the fleet size of one is the classic example.
- Early stages buy survival of one machine; later stages buy survival of one location. They cost very differently.
- Each stage adds a permanent meter and a permanent operational obligation, neither of which is removed by the next stage.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Stage 1 gives the workload an address and a supervisor: DNS points at a public IP, a process manager restarts the app, TLS terminates on the box.
- • Stage 2 separates the durable thing from the disposable thing, so an app-level fault can no longer destroy state.
- • Stage 3 puts a stable address in front of a replaceable fleet; the address survives instance replacement because it was never an instance address.
- • Stage 4 moves work that does not fit inside a request — large files, expensive reads, long jobs — off the request path.
- • Stages 5 and 6 make the whole shape reproducible and elastic; stage 7 duplicates it in a second failure domain and inherits the data problem that creates.
- • A certificate lifecycle from stage 3 onward, automated before the first expiry rather than after it.
- • Health-check semantics: what "healthy" means must include the dependencies the request path actually needs.
- • Backup *and* restore verification from stage 2 — the dump you have never restored is not a recovery plan.
- • A written record of which assumptions each stage invalidated, because that record is what onboarding and incident review both need.
- • The scheduled-job problem: from stage 3, anything that must run exactly once needs a leader, a lock or a queue.
- • Stage 1: the disk fills with logs, the database cannot write, and the application returns 500s while CPU and memory look perfectly healthy.
- • Stage 3: a deploy replaces both instances at once because the rollout had no surge or drain settings, and the "zero-downtime" architecture has a 30-second outage.
- • Stage 3: the load balancer health check probes
/which returns 200 from a static handler, so it keeps routing to an instance whose database connection pool is exhausted. - • Stage 4: the queue backs up and nobody notices, because the API stayed fast — the failure moved to a component with no user-facing latency signal.
- • Stage 6: autoscaling reacts in 90 seconds to a burst that saturates in 20, so every burst is served badly and the scale-out arrives for the trough.
- • Stage 7: a region failover works exactly as designed and loses the last 40 seconds of writes, because the replication was asynchronous and nobody had agreed an RPO.
- • Stage 1 to 3 is bounded by one machine; the first ceiling is usually database connections or file descriptors, not CPU.
- • Stage 4 removes the request-path ceilings and moves the bottleneck to the database, which is where it stays for most systems for a long time.
- • Stage 6 makes capacity elastic but introduces lag: new capacity is minutes late, so the real question becomes how much headroom you keep warm — see Startup Time & Cold Start.
- • Stage 7 does not increase capacity in a useful sense for most workloads; it changes the failure domain and the latency floor, and it makes the write path harder.
- • Stage 1 exposes a machine directly: SSH on the public internet and a database listening on all interfaces are the two findings that show up in every early audit.
- • Stage 3 is where the boundary becomes real: the load balancer is the only public thing, and everything else moves behind it — see Public and Private Subnets.
- • Stage 5 makes the pipeline a privileged identity. CI that can deploy can usually also destroy; that identity deserves the same scrutiny as a production admin — see The Pipeline as Infrastructure.
- • Stage 7 multiplies every secret, key and policy across regions, and a rotation that misses one region fails in the least convenient way possible.
- • The shape changes at every stage: fixed at 1–3, usage-shaped from 4, and non-linear in egress from 6.
- • Redundancy roughly doubles the compute line for the same throughput. That is the price of availability and should be stated as such in the design review.
- • Cross-region transfer at stage 7 is continuous, bidirectional in active-active designs, and usually the highest per-GB rate on the price list.
- • Human attention is the meter that caps the whole progression, and it is the only one that does not appear on an invoice.
- • From stage 1: is the process alive, is the disk filling, is the certificate expiring. Three checks that prevent most early outages.
- • From stage 3: per-target health and per-target error rate — an aggregate error rate of 3% hides one instance failing 100% of its requests.
- • From stage 4: queue depth and oldest-message age, which are the only signals that reveal a worker tier falling behind.
- • From stage 6: scaling events against saturation, so you can see whether capacity arrived before or after the burst.
- • The signal that lies at every stage: an aggregate success rate. It stays green while a meaningful minority of users fail completely.
- • Stop at stage 2 with a managed application platform. For a workload with a few thousand users and no strict availability target, a PaaS plus a managed database provides stages 3–5 as someone else's problem, and that is frequently the right answer for years.
- • Skip stage 6's autoscaling entirely and provision for peak. If your peak is 3x your mean and instances are cheap relative to engineering time, fixed capacity is simpler, faster under burst, and has no scaling-lag failure mode.
- • Replace stage 7 with a CDN and read replicas. Most "we need multi-region for latency" requirements are satisfied by caching static and read-heavy responses at the edge, without acquiring a distributed write problem.
- • For a purely static or mostly-static site, the entire progression collapses to object storage plus a CDN, and none of stages 1–7 apply.
- • Each stage buys a specific guarantee and charges a permanent operational obligation; the obligations accumulate and the guarantees do not compound.
- • Moving too early costs money and attention on a problem you do not have; moving too late costs an incident, and the incident is usually cheaper than three years of premature complexity.
- • Later stages make the system more available and less comprehensible. The number of people who can debug a stage-7 system at 03:00 is much smaller than for a stage-3 one.
From laptop to production
What people believe, and what is true
You should build the end state up front to avoid migrations.
You do not know the end state. Building stage 6 on day one means operating stage-6 complexity with stage-1 knowledge of the workload, and the parts you guessed wrong are harder to remove than they would have been to add.
Stage 7 makes the system highly available.
Multi-region changes the failure domain and creates a data-consistency problem. Availability comes from a tested failover with an agreed RPO and RTO, which most multi-region deployments do not have — see Multi-Region Deployment.
A load balancer means zero-downtime deploys.
It makes them possible. Actually achieving them requires connection draining, readiness checks that reflect dependencies, and a rollout that never removes all healthy targets at once.
Go deeper
Overview
A system grows by fixing one observed failure at a time. Laptop, one server, split the database, add redundancy, move work off the request path, automate, scale, and — rarely — replicate to a second region.
Practical
For your own system, write the current stage and the specific failure that would force the next one. If you cannot name the failure, you are at the right stage.
Advanced
Track what each stage let you *stop* assuming, and what it made newly assumable. Stage 3 stops "one instance" and starts "any instance can serve any request", which is only true if session, file and scheduling assumptions were fixed. Most stage-3 incidents are unfixed stage-2 assumptions.
Internals
The progression is a sequence of failure-domain subdivisions. Stage 1 has one domain containing everything. Stage 2 splits state from compute. Stage 3 splits compute into two, and the load balancer is a domain-aware router. Stage 6 makes the domain count elastic. Stage 7 adds a domain that cannot share memory or a synchronous clock with the others, which is why it — and only it — forces a distributed-consistency decision. See Failure Domains.