Why Stateful Workloads Are Harder
A stateless replica is interchangeable and can be replaced at any moment. A database replica has an identity, a copy of the data, a position in a replication stream and an opinion about who is the leader.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why can I replace an API pod at will and not a database pod?
Every operational technique that makes stateless services easy — replace on failure, scale by adding, roll out in any order, treat instances as interchangeable — depends on the assumption that instances hold nothing that matters. Data breaks all four.
Treat the database like any other workload: give it replicas, attach storage, put a Service in front, and let the orchestrator handle failure the way it handles everything else.
The replicas are not interchangeable. One accepts writes and the others do not, and the Service in front of them has no idea which is which (Services: A Stable Address Over Moving Pods).
- The replicas are not interchangeable. One accepts writes and the others do not, and the Service in front of them has no idea which is which (Services: A Stable Address Over Moving Pods).
- Replacing a replica is not free. A new member has to obtain a copy of the data, catch up on the replication stream, and only then serve — which can take minutes or hours, not seconds (Replication and Read Scaling).
- Rolling in arbitrary order is unsafe. Restarting the leader mid-rollout triggers a failover; restarting two members at once can lose the quorum that allows writes at all.
- Scaling out does not add capacity for writes. Adding read replicas adds read capacity and adds replication lag, which is a correctness question for the application, not a capacity one (Replication Lag: Reads That Are Correct and Stale).
- Storage ties each replica to a place. The pod can only run where its data can attach, so "reschedule it elsewhere" is not always available (Volumes: Storage With a Lifecycle).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Four properties separate stateful from stateless, and every one of them is an assumption the stateless model quietly makes.
- Identity: a member is addressable and known to its peers. Losing and recreating it under a different name is not the same member returning; it is a new member the cluster has to admit.
- Durable storage bound to identity: member 2 must come back with member 2's data. Attaching a fresh empty disk creates a member that must be rebuilt from scratch.
- Ordering: startup, shutdown and upgrade order matter. Leaders must be handled deliberately, and quorum-based systems will refuse writes rather than risk divergence if too many members are down at once.
- Replication semantics: whether a write is acknowledged before or after replicas have it decides what a failover costs. Asynchronous replication means failover can lose recent writes; synchronous means a slow replica slows every write (Distributed Consistency: CAP, Quorums, Consensus).
- None of this is caused by Kubernetes and none of it is fixed by Kubernetes. It is what data costs, and it was equally true on VMs — the orchestrator just makes it easier to accidentally treat the workload as if it were not.
The four assumptions stateless services get for free
Every row is something the stateless deployment model assumes without stating it. Reading the right column tells you which operational practice has to be replaced rather than reused.
| Assumption | Stateless service | Data system | What has to change |
|---|---|---|---|
| Replicas are interchangeable | Any replica serves any request | One leader accepts writes; readers may be stale | Separate write and read addressing; do not select by one label |
| Replacement is instant | Pull image, start, serve | Obtain a data copy, catch up, then serve | Capacity planning must survive a member being absent for a long time |
| Order does not matter | Roll in any order | Leader last; never break quorum | Ordered rollout with health gating between members (StatefulSets: Identity, Storage and Order) |
| Scaling adds capacity | More replicas, more throughput | More replicas add reads and replication lag, not writes | Sharding or a bigger primary; not a replica count change (Partitioning and Sharding) |
What a failover actually costs
Failover is where all four properties arrive at once. The sequence below is a healthy, well-run failover — it still costs a window of unavailability and, depending on replication mode, some number of acknowledged writes.
Knowing this sequence is what lets you answer the question that matters during an incident: is promoting a replica right now better or worse than waiting for the primary to come back?
- 1Detect
Decide the primary is genuinely unavailable rather than briefly slow.
fails by Too eager and you fail over on a network blip; too slow and the outage is the detection window.
evidence Detection threshold is written down and matches observed transient durations.
- 2Choose
Pick the replica with the most complete replication position.
fails by Promoting a lagging replica discards the writes it never received (Replication Lag: Reads That Are Correct and Stale).
evidence The chosen replica's position is recorded before promotion.
- 3Fence
Ensure the old primary cannot accept writes if it returns.
fails by Skipping this is how split brain happens, and it is discovered days later.
evidence The old primary is demoted or isolated before the new one is promoted.
- 4Promote
Make the chosen replica the writer.
fails by Promotion succeeds while the application still points at the old address (Service Discovery in Operation).
evidence Writes succeed against the new primary from an application host.
- 5Redirect
Point applications at the new writer.
fails by Every instance reconnects simultaneously and saturates the new primary (The Connection Budget).
evidence Connection counts settle and query latency returns to baseline.
- 6Rebuild
Return the old primary as a replica, from a copy or from the stream.
fails by Rebuilding during peak load competes with live traffic for I/O.
evidence Replica count is back to normal and lag is within threshold.
The unavoidable cost is between Detect and Redirect. Everything after that is capacity work; everything before it is the outage.
Deciding where the data lives
This is the decision that determines how much of the preceding lesson is your problem. There is no default answer, and the most common mistake is not choosing at all — inheriting a decision from a tutorial.
You need a relational database for a service running in Kubernetes. Where does it go?
when The default for most teams. You need a database, not a database operations practice.
cost Cost scales with usage, version and extension choices are constrained, and failover behaviour is the provider's design, not yours.
when You have a genuine constraint — data residency, cost at scale, an unsupported engine — and someone who owns the operator's behaviour under failure.
cost Backups, upgrades, failover and storage performance become yours, plus the operator itself as a component to understand (StatefulSets: Identity, Storage and Order).
when Development, tests, and workloads whose data is genuinely reconstructible.
cost Every mechanism in this lesson is yours to implement. Reasonable for a cache; a serious commitment for a system of record.
when Performance-sensitive systems where you want the hardware and the tuning to be predictable.
cost A second operational model to maintain alongside the cluster, and no shared tooling for deploys or access.
How to do it properly
Most important first.
- Start by asking whether you should be operating this yourself. A managed database removes the failover, backup, upgrade and replication work, and that is most of the work (Managed Databases).
- If you do run it, run it as the thing it is: stable identity per member, storage bound to that identity, deliberate ordering, and a failover procedure that has been rehearsed rather than reasoned about (StatefulSets: Identity, Storage and Order).
- Separate the roles at the network layer. Writers and readers need different addresses, because "the current leader" is not something a label selector can express.
- Know your replication mode and what a failover costs in lost writes. That number is your real recovery point objective, whatever the documentation says (RTO and RPO).
- Back up independently of the replication. Replication copies your mistakes faithfully; a dropped table arrives on every replica in milliseconds (Backup Operations).
- Rehearse the failure. A failover you have performed deliberately is a procedure; one you have only read about is a hypothesis (Restore Drills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Data loss and split brain are not contained by any deployment mechanism; the only real containment is a tested restore and a replication mode whose worst case you have accepted in advance.
What can go wrong
- Split brain: two members both believe they can accept writes, and the divergence is discovered later as inconsistent data with no clean merge.
- Failover to a replica that was behind, silently discarding the writes it had not received. The system is available and has lost data (Replication Lag: Reads That Are Correct and Stale).
- Rolling restart that takes down enough members to break quorum, so the whole system stops accepting writes during what was planned as a routine upgrade.
- A recreated member coming up with an empty volume and syncing a full copy at exactly the moment the system was already under stress.
- Backups that exist and have never been restored, so the recovery time is unknown until it is being measured during an incident (Restore Drills).
- Connection storms after a failover: every application instance reconnects at once, and the new leader is saturated by connection setup before it serves a query (The Connection Budget).
- "Stateless is better than stateful." The state has to live somewhere. Stateless services are easy precisely because something else is carrying the hard part (Pets and Cattle, Read Carefully is the dogma to avoid here).
- "Replication is a backup." Replication copies every write, including the destructive ones, immediately (Backup Operations).
- "More replicas means more availability." More replicas means more members that must agree, and in quorum systems it changes how many failures you can tolerate rather than simply improving on it.
- "The orchestrator handles failover." It restarts and reschedules processes. Which member is the leader, and whether a promotion is safe, is the data system's decision or yours.
- "This is a Kubernetes problem." It is a data problem. Kubernetes makes it easier to run into and no harder to solve.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A deliberate failover has been performed in production or a production-like environment, with a recorded duration and a recorded number of lost or delayed writes.
- Replication lag is monitored continuously with an alert threshold that reflects what the application can tolerate.
- A restore into a fresh instance has been completed and timed within the last quarter (Restore Drills).
- Each member's identity maps to its own storage, verified by deleting a member and watching it return with its data rather than syncing from empty.
- You largely cannot roll back state. A write that happened, happened; a schema change that ran, ran. This is the one place in the domain where "you cannot, and that is the point" is the honest answer (Destructive Migrations).
- What you can do is design changes to be reversible before you make them — expand before you contract, keep the old column, keep the old format readable (Expand, Migrate, Contract).
- The genuine recovery path for data is a restore, and its recovery point is whenever the last backup was taken (Partial and Logical Data Recovery).
- Automate backups, backup verification and a periodic restore. This is the highest-value automation in the whole area, and the restore is the part that matters (Backup Operations).
- Automate replication lag and quorum health monitoring, since both are invisible from the application side until they are severe.
- Automate failover only if you trust the detection. An automated failover on a false positive costs you writes and gains nothing (The Automation Trap).
- Keep the decision to fail over a region, or to restore over live data, human. Those are judgement calls with unrecoverable outcomes (Region Failover).
- Managed services remove most of this work and give up control over version timing, extensions, tuning and failover behaviour, plus a cost that grows with usage.
- Synchronous replication protects recent writes and makes every write as slow as the slowest replica.
- Running data systems next to your workloads gives you locality and one platform to operate, at the cost of every failure mode in this lesson being yours (StatefulSets: Identity, Storage and Order).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICWhat a cluster adds is the temptation to treat a data system like every other workload: one Service selecting all replicas, rescheduling on node failure, and a rolling restart with no notion of leadership. A VM fleet forces you to name each member and touch it deliberately, which is slower and much harder to do accidentally.
- GENERALIdentity, durable storage, ordering and replication semantics are properties of distributed data systems, not of any platform. They were true of a database on bare metal and remain true on a managed service — what changes is who operates them.
- DATABASE-SPECIFICWhat a failover costs depends entirely on the engine and its configuration: a synchronously replicated cluster loses nothing and pays latency on every write; an asynchronous one is faster and can lose whatever had not shipped. The engine's locking and quorum rules also decide whether a rolling restart is routine or an outage.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — quorum, consensus and the reason a majority must agree before a write is safe.