Blue-Green Deployments
Run two complete environments, cut traffic from one to the other, and cut back if it is wrong — while remembering that the database was never duplicated.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
When is it worth paying for two full environments to make rollback instant?
A bad release must be reversible in seconds, not in the time it takes to roll a fleet back one batch at a time.
Stand up the new version as a second full environment, point the load balancer at it, and keep the old one around for a while in case we need to switch back. Instant deploy, instant rollback.
The switch is instant for traffic and not for data: green has already written rows in the new format, so switching back to blue puts old code in front of data it cannot read.
- The switch is instant for traffic and not for data: green has already written rows in the new format, so switching back to blue puts old code in front of data it cannot read.
- Traffic is cut by changing DNS, and clients keep resolving the old address for as long as their resolvers cache it — the "instant" switch has a long tail nobody controls (Following One Lookup Through Every Cache in Networking).
- Green starts cold: empty caches, empty pools, unwarmed runtime. Cutting 100% of traffic to it at once produces a latency spike and possibly a thundering herd on the database (Cache Stampede).
- In-flight requests on blue are killed at the moment of the switch, because nobody drained it (Graceful Shutdown).
- Both environments run queue consumers, so during the overlap jobs are processed by both versions in an order nobody planned.
- The environment costs double while both exist, and "keep blue around for a while" quietly becomes permanent.
What is actually happening
- Two complete environments — blue (live) and green (idle or newly deployed) — differ only in version. A routing change moves traffic from one to the other, and the previous environment stays intact as a rollback target.
- The switch happens at a routing layer: load balancer target group, service selector, reverse-proxy upstream, or DNS. The layer determines how fast the switch really is, and DNS is by far the slowest and least controllable.
- What is duplicated is compute. The database, the cache, the object storage and the queues are almost always shared, because duplicating them would mean duplicating state and reconciling it.
- That asymmetry is the entire subject. Rollback is instant for code and impossible for data. Anything green wrote persists after the switch back.
- Because both environments talk to one database, every compatibility requirement from a rolling deploy still applies — with the twist that the overlap is short but the rollback window is long.
- Draining matters at the cut: blue must finish in-flight requests after it stops receiving new ones, exactly as in a rolling deploy.
What is duplicated, and what never is
Every blue-green diagram shows two neat stacks. The honest version has one box at the bottom shared by both, and that box is where the risk lives. Compute is duplicated because it is stateless; state is not duplicated because reconciling two copies of it is a harder problem than the one you were solving.
This makes the rollback guarantee asymmetric in a way worth stating plainly: traffic rollback is instant; data rollback does not exist. Every schema and data-format decision must therefore be compatible with both versions for as long as blue is a real rollback target.
Cutting over without a cliff
A hard cut moves 100% of traffic to an environment that has never served any. Its connection pools are empty, its caches are cold, its runtime is unwarmed, and every cache miss it generates lands on the database at the same moment. The result looks like a performance regression in the new version and is actually a temperature problem.
The fix is to make the cut a short ramp. Even a two-minute ramp through 5% and 25% builds pools, warms caches and gives the health signals something to say before the old environment is idle.
- 1Deploy green
Full environment, same config as blue except version.
fails by Config drift — the tested environment differs from the live one.
- 2Verify green privately
Health, startup validation, synthetic requests on an internal address.
fails by Only checking that the process is up (Health Checks: Startup, Readiness, Liveness).
- 3Warm green
Small real traffic share or synthetic load to build pools and caches.
fails by Skipping it, producing a latency spike blamed on the new code.
- 4Ramp traffic
5% -> 25% -> 100% with a pause and a look at per-version signals.
fails by A hard cut, which removes every decision point.
- 5Drain blue
Stop routing, let in-flight requests finish, keep the process alive.
fails by Terminating blue at the cut, killing in-flight requests.
- 6Move background work
Consumers and schedulers activate in green, deactivate in blue.
fails by Both active: jobs run twice (Job Idempotency).
- 7Hold as rollback target
Blue idle, patched, reachable only internally, for a decided window.
fails by Holding it forever, doubling cost and leaving an unmonitored production deployment.
- 8Decommission
Blue removed; green becomes the new blue.
fails by Decommissioning before the data written by green is proven good.
The rollback that does not roll back
The scenario worth rehearsing: green ships, runs for forty minutes, and is found to be corrupting a field. Traffic is cut back to blue in seconds and the incident is declared over — except forty minutes of writes are in the new format, and blue is now serving requests against them.
This is why blue-green and expand-contract are not alternatives. The instant rollback is only genuinely instant when the previous version can read everything the new one wrote, which is exactly the property expand-contract exists to preserve.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Cut to green | Latency spike for a few minutes, then normal | Cold pools and caches; database absorbing a miss storm | Warm green with a ramp before the full cut (Cache Stampede). |
| Cut to green | A burst of connection resets at the switch | Blue terminated instead of drained | Stop routing first, drain, then idle the environment. |
| Cut to green | Some clients still hitting the old version for hours | Switch made at DNS; resolver caches ignore TTL | Switch at the load balancer; treat DNS as a last resort. |
| Rollback to blue | Errors on records created in the last hour | Green wrote a format blue cannot read | Expand-contract, and a compensating backfill for the affected window. |
| Overlap period | Duplicate emails, double-charged jobs | Both environments running consumers and schedulers | One active environment for background work, enforced by config or leader election. |
| Overlap period | Database refusing new connections | Two environments' pools summing over the limit | Size pools for the overlap, not for one environment (Connection Pools). |
| Days later | An old, unpatched environment is still reachable | No decommission deadline | Enforce a rollback window with an alert on expiry. |
How to build it
Most important first.
- Switch at a layer you control synchronously — a load balancer target weight or a proxy upstream — not at DNS. If DNS is the only option, lower the TTL well in advance and expect a tail regardless.
- Keep the database compatible with both environments for the whole rollback window. Blue-green does not replace expand-contract; it makes it more important, because rollback is expected rather than exceptional (Expand and Contract Migrations).
- Warm green before the cut: send synthetic traffic or a small real share first so pools are built and caches are populated. A blue-green with a brief canary phase is strictly better than a hard cut.
- Drain blue rather than terminating it: stop routing, wait for in-flight requests, then leave it idle as the rollback target (Graceful Shutdown).
- Decide explicitly what happens to background workers and schedulers. Usually only one environment should consume queues and run scheduled jobs; a leader election or a config flag decides which (Scheduled Jobs).
- Set a deadline for decommissioning blue, and enforce it. The rollback window is a decision, not an accident of nobody cleaning up.
- Write down what "rollback" means for data: which changes are reversible by switching, and which require a compensating action.
What can go wrong
- Rollback taken after green has written incompatible data — the switch succeeds and the service is broken in a new way.
- Green environment drifts from blue in configuration, so the environment that was tested is not the environment that goes live.
- Both environments consuming the same queue, so jobs are processed twice or by the wrong version (Job Idempotency).
- Scheduled jobs firing in both environments — the classic double-billing bug.
- A hard cut to a cold environment causing a self-inflicted load spike on the database as every cache miss arrives at once.
- Long-lived connections (WebSockets, SSE, gRPC streams) never move, because they were established against blue and nothing forces them to reconnect (Load Balancing, From the Backend's Side).
- At the moment of the cut, in-flight requests exist on blue while new requests arrive on green — both writing to the same rows.
- If both environments run queue consumers or schedulers, the same job can be claimed by both; only one should be active, enforced by config or leader election.
- A client retry can cross the cut: the first attempt handled by blue, the retry by green. Both must be idempotent (Idempotency in Backends).
- The idle environment is still a running, reachable, and often less-monitored deployment holding production credentials. Restrict its network exposure and keep it patched, or decommission it (Public Exposure, Read With Context in Cloud & Infrastructure).
- A security fix is not fully deployed while blue is still routable. Cut traffic and then remove the old environment, rather than leaving a vulnerable version warm.
- Two environments means two sets of instance credentials and two audit trails. Ensure logs from both are collected under the same identity model (Audit Logs for Privileged Actions in Security Engineering).
- "Blue-green means we can always roll back." You can always roll back *traffic*. Data written by green survives the rollback, and that is usually what makes a bad release bad.
- "Blue-green avoids the two-version problem." It shortens the serving overlap. Both versions still share a database, and the rollback window means old code must stay compatible for longer, not shorter.
- "It is zero downtime by construction." Only with draining, warmed pools and a switch layer that is actually synchronous. A DNS cut with no drain is downtime distributed across clients.
- "Blue-green is safer than rolling." It is safer against *bad code* and identical against *bad schema changes* — and it costs double capacity to get there.
Operating it
- Label everything by environment as well as version, so the graphs during the cut are readable rather than a single confused line.
- Watch connection counts to the database from both environments: during the overlap the total is the sum, which can exceed the connection limit (Connection Pools).
- Track the time from switch to steady state — cache hit rate recovering and latency settling is the real end of the deploy, not the routing change.
- Alert on an idle environment that has existed longer than the intended rollback window.
- Cost scales with the size of the environment: doubling ten instances is easy, doubling four hundred is a capacity conversation with a real number attached.
- The cold-start effect grows with scale, because a larger fleet warming simultaneously puts a larger simultaneous load on shared dependencies.
- At scale, blue-green tends to converge toward canary anyway — teams start shifting traffic in increments rather than cutting it, which is a canary with two environments.
- Instant traffic rollback is bought with roughly double the compute for the duration of the overlap, plus the operational cost of keeping two environments identical.
- A hard cut avoids a long two-version window and produces a cold-start spike instead. A gradual cut avoids the spike and reintroduces the window.
- Keeping the rollback target alive longer increases safety and cost simultaneously, and there is no correct number — only a decided one.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe pattern — duplicate compute, switch routing, keep the previous target — is platform-independent.
- CLOUD-SPECIFICThe switch layer differs and so does its speed: a load-balancer target-group swap or a service-selector change is effectively immediate, while a DNS record change is bounded by resolver caching that ignores your TTL more often than the specification suggests.
- SCALE-SPECIFICDoubling compute is a rounding error for a small service and a serious capacity request for a large one. Above a certain fleet size teams shift to incremental traffic moves, which is canary in blue-green clothing.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.