The question this answers
What changes when a workload owns data, and should that workload be running in the cluster at all?
PostgreSQL must survive pod restarts, node failures and cluster upgrades without losing a committed transaction, must be restorable to a point in time, and must have exactly one writer at any moment.
Stable network identity per replica, a persistent volume that follows that identity across rescheduling, and ordered creation and deletion — the minimum a replicated data system needs from a scheduler.
Four properties a stateless workload does not have
Everything comfortable about a Deployment comes from replicas being interchangeable. Any pod can serve any request, order does not matter, a replacement is as good as the original, and traffic goes to whichever is Ready. A database has none of those properties, and each one it lacks is a feature a StatefulSet adds.
Identity: the replicas are not equivalent. db-0 is the primary; db-1 and db-2 are followers with specific positions in the replication stream. A StatefulSet gives each pod a stable ordinal name and a stable DNS record, so db-1 restarting is still db-1. Storage: each replica owns a volume, and the replacement pod must attach *that* volume, not a fresh one. Ordering: db-0 comes up before db-1 and goes down after it, because a replica cannot join a cluster whose leader does not exist. Membership: adding a replica means seeding it from a backup or a base copy and having it catch up — a process measured in minutes to hours, not in a scheduler decision.
Note what a StatefulSet does *not* give you. It does not do failover. It does not promote a follower when the primary dies. It does not know what replication is. It provides identity, storage binding and ordering; every actual database behaviour — election, promotion, backup, point-in-time restore — comes from an operator you install or a runbook a human executes at 03:00. That gap is the entire cost of this lesson.
| Question | Stateless API (Deployment) | Database (StatefulSet) |
|---|---|---|
| Are replicas interchangeable? | Yes — any pod serves any request. | No. One is the writer; the others are behind it by some amount of replication lag. |
| What happens on replacement? | A new pod with a new IP starts fresh and serves immediately. | The replacement must reattach the same volume and rejoin at the right stream position. |
| Does startup order matter? | No. Start all replicas at once. | Yes. The leader must exist before followers can join. |
| What is lost if a pod dies? | In-flight requests. | Potentially committed data, if replication was asynchronous and had lag. |
| Can it be scaled by changing a number? | Yes, in seconds. | No. A new replica must be seeded and caught up first. |
| Who handles failover? | The Service — it stops routing to unready pods. | An operator, or a human. Kubernetes will not promote anything. |
Where the volume actually lives, and why zones matter
A persistent volume is a real disk provisioned by the provider, and in most clouds a block volume is zonal: it exists in one availability zone and can only be attached to a machine in that zone. That single fact drives most stateful scheduling surprises. If db-0's volume is in zone A, db-0 can only ever be scheduled onto a node in zone A. If zone A is unavailable, db-0 is unschedulable — not slow, not degraded, unschedulable — and the pod sits Pending until the zone returns.
This is why zone redundancy for a database is a *replication* design, not a scheduling one. Three replicas across three zones survive a zone loss only if the data is replicated between them and something can promote a survivor. Three replicas in one zone with three volumes in that zone survive nothing. The cluster will happily let you build the second one.
The other trap is the reclaim policy. A volume claim outlives the pod by design, and depending on configuration it can also outlive the StatefulSet — which is good, because deleting a workload should not delete its data, and expensive, because orphaned volumes keep billing silently for months. Both halves of that sentence are true and both surprise people.
The honest recommendation: usually, do not
Running a database on Kubernetes is a legitimate engineering choice, and for most teams it is the wrong one. The reason is not that it does not work — mature operators genuinely handle election, backup and failover well. The reason is that it moves a very large, very specialized operational burden from a provider that runs thousands of these to a team that runs one, and it does so for a saving that is smaller than it looks.
A managed database gives you patching, backups, point-in-time restore, automated failover, replica provisioning, version upgrades and someone to call. Doing all of that yourself means owning storage performance, backup verification, upgrade rehearsal, failover testing and an on-call rotation with people who understand replication. That is a team's worth of expertise being spent to avoid a bill that is often a small fraction of the salary it costs.
There are real reasons to run it in-cluster: a hard data-residency or air-gap requirement, a database with no managed offering, an extension the managed service does not permit, cost at a scale where the difference funds a dedicated team, or a workload where the data is genuinely rebuildable. If one of those applies, use a mature operator rather than raw StatefulSets — and put the same rigor into restore rehearsal that you would demand of a provider. If none applies, put the database outside the cluster and spend the attention elsewhere. See Managed vs Self-Hosted for the full comparison.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- A StatefulSet supplies identity, stable storage binding and ordering — not failover, promotion, backup or replication.
- Block volumes are usually zonal, so a replica is pinned to the zone its volume lives in; a zone outage makes that pod unschedulable, not slow.
- Zone redundancy for data is a replication design, and the cluster will let you build three replicas in one zone that survive nothing.
- Volume claims outlive pods by design, which protects data and quietly bills for orphaned disks.
- A managed database is usually the better answer. Self-hosting trades a modest bill for a specialized operational burden most teams should not take on.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A StatefulSet creates pods with stable ordinal names and a stable DNS record each, backed by a headless Service.
- • Each pod gets its own persistent volume claim, which binds to a real provider disk and follows that ordinal across rescheduling.
- • Pods are created in order and terminated in reverse order, so a leader exists before followers start and outlives them on shutdown.
- • The scheduler must place each pod in the zone where its volume lives, which constrains placement far more than resources do.
- • Everything above the scheduler — election, promotion, replication, backup — is implemented by an operator or by humans, not by Kubernetes.
- • Backups, and restores you have actually performed into a real environment — see Restore Testing and Backup Strategy.
- • Version upgrades, which for a database are a rehearsed migration rather than a rolling image change.
- • Failover behaviour: what promotes a replica, how long it takes, and how the application discovers the new primary.
- • Storage performance: IOPS and latency characteristics of the volume class, which determine commit latency more than CPU does.
- • Connection management from the application side, since pods restarting means connection pools reconnecting in bursts.
- • A zone outage leaves a replica's volume unreachable; the pod is Pending indefinitely because no node in another zone can attach that disk.
- • Split brain: two pods believe they are primary after a partition, and writes diverge. Kubernetes has no opinion about this.
- • A volume that fills up: the database stops accepting writes, and a persistent volume does not grow on its own unless expansion is configured.
- • A rolling update applied to a StatefulSet without operator awareness, restarting the primary and causing an unplanned failover mid-deploy.
- • A backup that has never been restored, discovered to be unusable on the one day it is needed.
- • Read replicas scale reads and do nothing for writes; the single writer is the ceiling, and that is a data-model problem — see Managed Databases.
- • Adding a replica requires seeding and catch-up, so scale-out is measured in minutes to hours, not seconds. Autoscaling does not apply.
- • Storage IOPS is commonly the first hard limit, and on many providers IOPS is a function of provisioned size, so you grow the disk to buy throughput.
- • The volume is where the data actually is: encryption at rest is a property of the volume and its snapshots, not of the pod.
- • Database credentials in the cluster inherit every limitation in ConfigMap vs Secret — and the Honest Limit of a Secret; short-lived credentials are worth more here than anywhere else.
- • A database should never be exposed outside the cluster network. A public load balancer on 443 is a design; a database port reachable from the internet is the finding — see Public Exposure, Read With Context.
- • Backups are a full copy of your most sensitive data, and they are frequently the least protected artifact in the system.
- • Anyone who can exec into a database pod has the data. That is a much larger group than the people you granted database access to.
- • Storage is billed on provisioned capacity per replica, so three replicas are three full copies whether or not they are full.
- • Cross-zone replication traffic is metered on most providers and runs continuously for a write-heavy workload.
- • Orphaned volume claims outlive their workloads and bill indefinitely — a recurring, invisible line item.
- • The dominant cost is engineering time for upgrades, failover testing and restore rehearsal, and it is absent from every comparison spreadsheet.
- • Replication lag, which is the direct measure of how much data a failover would lose.
- • Volume free space and IOPS saturation, both of which fail abruptly rather than gradually.
- • Backup success *and* restore rehearsal results — a green backup job is not evidence of recoverability.
- • Pod scheduling status for the stateful workload, because Pending here means a zone or volume problem rather than a capacity one.
- • The signal that lies: pods Running. A database can be Running, Ready and hours behind on replication.
- • A managed database outside the cluster, reached over private networking. This is the right answer for most teams and most workloads.
- • A managed database with the application in the cluster — a very common and very sensible split that keeps the hard part with the provider.
- • A mature operator rather than raw StatefulSets, if you must self-host: election, backup and failover are what you are actually buying.
- • For rebuildable data — caches, search indexes, derived stores — in-cluster is genuinely fine, because losing a replica costs a rebuild rather than a restore.
- • Buys stable identity and storage for workloads that need them; costs placement constraints that turn a zone outage into an unschedulable pod.
- • Buys one deployment mechanism for everything you run; costs owning database operations that a provider would otherwise own.
- • Buys lower compute and storage line items; costs specialized on-call expertise that is more expensive than the saving.
What people believe, and what is true
A StatefulSet makes a database highly available.
It provides identity, storage and ordering. Election, promotion and replication come from an operator or a human; Kubernetes will not promote a replica.
Persistent volumes follow the pod anywhere in the cluster.
On most providers a block volume is zonal. The pod follows the volume, not the other way around, and a zone outage makes it unschedulable.
Running our own database saves money.
It saves on the invoice and spends on people. Upgrades, failover testing, restore rehearsal and on-call depth usually cost more than the managed premium.