K8s StateKUBERNETES-SPECIFICDATABASE-SPECIFIC

StatefulSets: Identity, Storage and Order

The workload controller that gives each replica a stable name, its own volume and a defined position in startup and rollout — which makes running stateful systems possible, not advisable by default.

What actually happensHow to do it properlyEvidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What does Kubernetes offer a workload that needs stable identity, and what does it still leave to you?

The problem

Deployments deliberately make replicas interchangeable: random names, shared template, no ordering, one shared volume at most. Data systems need the opposite of all four.

What teams do first

Swap kind: Deployment for kind: StatefulSet and the workload is now safe to run in the cluster, because StatefulSet is the object for stateful things.

How it breaks

It gives you identity, storage and ordering. It does not give you leader election, replication configuration, backups, failover, or an upgrade path that understands your engine (Why Stateful Workloads Are Harder).

How it breaks in production
  • It gives you identity, storage and ordering. It does not give you leader election, replication configuration, backups, failover, or an upgrade path that understands your engine (Why Stateful Workloads Are Harder).
  • Ordering is about pod lifecycle, not about data. It restarts members one at a time; it has no idea which one is the leader or whether restarting this one breaks quorum.
  • The per-replica volumes it creates are not deleted when you scale down. That is deliberate and correct, and it means scaling down leaves real disks behind that you must reason about.
  • Recovery gets slower rather than faster. A replaced member reattaches its old volume and must catch up, and if the volume is gone it rebuilds from scratch at the worst possible moment.
  • The kubectl ergonomics stay stateless-shaped. Deleting a pod is still trivial and now potentially means a failover.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

  • Pods get stable ordinal names — db-0, db-1, db-2 — that survive rescheduling. A replaced db-1 is still db-1, at the same DNS name, which is what peer-to-peer membership needs.
  • volumeClaimTemplates create one claim per replica, bound to that ordinal. db-1 always gets db-1's data, wherever it runs (Volumes: Storage With a Lifecycle).
  • A headless Service gives each pod its own stable DNS record, so members can address each other individually rather than through a load-balanced virtual IP (Services: A Stable Address Over Moving Pods).
  • Ordering is enforced: by default pods start in ordinal order and each waits for the previous to be Ready, and they terminate in reverse. podManagementPolicy: Parallel disables this for workloads that do not need it.
  • Rolling updates proceed in reverse ordinal order, one pod at a time, waiting for readiness. updateStrategy.rollingUpdate.partition holds ordinals below a threshold back, which is how a staged upgrade of a cluster member is expressed.
  • Everything above is lifecycle mechanics. Which member is the leader, when a promotion is safe, and how a new member joins are the data system's concern — which is what an operator, a purpose-built controller for a specific engine, exists to encode.

What changes when you switch controller

Four differences, and each one exists because a data system needs the opposite of what makes stateless replicas easy.

PropertyDeploymentStatefulSetWhy the difference exists
Pod namesRandom suffix, changes on replaceStable ordinal: db-0, db-1Peers address each other by name and must survive replacement
StorageOne shared claim, or noneOne claim per replica, bound to the ordinalMember 1 must come back with member 1's data
Network identityOne Service address for all podsPer-pod DNS via a headless ServiceReplication targets individual members, not a balanced address
OrderAny order, in parallelOrdinal order, one at a time, gated on readinessStarting or stopping too many members at once breaks quorum

The manifest, and the two fields that are safety controls

Most of this is unremarkable. volumeClaimTemplates and updateStrategy.rollingUpdate.partition are the two that change what happens on a bad day.

A three-member StatefulSet with a staged update
1apiVersion: v1
2kind: Service
3metadata:
4  name: db-peers
5spec:
6  clusterIP: None            # headless: per-pod DNS, no virtual IP
7  selector:
8    app: db
9  ports:
10    - name: peer
11      port: 5432
12---
13apiVersion: apps/v1
14kind: StatefulSet
15metadata:
16  name: db
17spec:
18  serviceName: db-peers      # the headless Service providing per-pod DNS
19  replicas: 3
20  podManagementPolicy: OrderedReady   # default: start 0, then 1, then 2
21  updateStrategy:
22    type: RollingUpdate
23    rollingUpdate:
24      partition: 2           # only ordinals >= 2 update: db-2 is the canary
25  selector:
26    matchLabels:
27      app: db
28  template:
29    metadata:
30      labels:
31        app: db
32    spec:
33      terminationGracePeriodSeconds: 120
34      containers:
35        - name: db
36          image: registry.example.com/db@sha256:41ab7d...
37          volumeMounts:
38            - name: data
39              mountPath: /var/lib/data
40  volumeClaimTemplates:
41    - metadata:
42        name: data
43      spec:
44        accessModes: ["ReadWriteOnce"]
45        storageClassName: standard-zonal
46        resources:
47          requests:
48            storage: 100Gi

Pods become db-0, db-1, db-2, each with its own claim named after it. partition: 2 upgrades only db-2, so you can verify one member before lowering the partition to roll the rest — a canary expressed in the controller rather than in a pipeline (Canary: One Percent, Then Five, Then Watch). The claims are not deleted when replicas is reduced.

What it does for you, and what it still leaves you

KUBERNETES-SPECIFICThe split between lifecycle and engine knowledge is what operators exist to close, and it is a Kubernetes-specific pattern. A managed database has no equivalent seam — the provider owns both halves, which is precisely what you are paying for.

This split is the whole lesson. The left column is genuinely valuable and is also the smaller half of running a data system.

Everything on the right is why an operator for your specific engine — or a managed service — is usually the right answer rather than a hand-written manifest.

The boundary of the abstraction
What the controller handles
Stable names across rescheduling. One volume per replica, bound to its ordinal. Ordered startup, ordered shutdown, one-at-a-time updates gated on readiness, and a partition field for staging them.
What remains entirely yours
Which member is the leader and when promotion is safe. How a new member joins and syncs. Replication mode and its data-loss window. Backups, and a restore you have performed. Engine version upgrades and their on-disk format changes. Connection behaviour after a failover.

The left column is lifecycle; the right column is the data system. Calling the left column "Kubernetes supports stateful workloads" is how teams end up operating a database they have never failed over. The honest framing is that StatefulSets make it *possible* to run one well, and an operator or a managed service is what makes it *likely* (Why Stateful Workloads Are Harder).

How to do it properly

Most important first.

  • Decide first whether the data should be in the cluster at all. For a system of record, a managed service is the default and StatefulSet is the exception that needs a reason (Managed Databases).
  • If you run a data system in-cluster, use a mature operator for that engine rather than a hand-written StatefulSet. The operator is where failover, backup and version upgrade knowledge lives.
  • Pair it with a headless Service, and add separate Services for the roles that clients actually need — a writer address and a reader address — because ordinal names are not roles.
  • Set a pod disruption budget so that voluntary disruptions — node drains, cluster upgrades — cannot take out enough members to break quorum.
  • Treat scale-down as a data operation. Removing a member is a membership change in the data system first and a replica count change second.
  • Back up from the data system's own mechanism, not from volume snapshots alone, unless you have restored a snapshot and confirmed the result is consistent (Backup Operations).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone
One testEveryone
What contains it

A change here touches the system of record; containment is the partition field for staged rollout, a disruption budget for quorum, and a restore path that has actually been exercised.

What can go wrong

Failure modes, including of the mitigation
  • Ordered startup stuck on db-0, so nothing else starts. Correct behaviour, opaque symptom: the fleet appears frozen with one unready pod.
  • A rolling update that restarts the leader, triggering a failover mid-upgrade with no one expecting it.
  • Scale-down that removes a voting member and leaves the remainder without quorum, so the system is read-only until someone notices.
  • Orphaned claims after scale-down, quietly billed, and then reattached with stale data when the workload scales back up.
  • Volumes and pods in different zones, so a member cannot be rescheduled after a node failure and stays Pending (Volumes: Storage With a Lifecycle).
  • A hand-written StatefulSet with no operator, where the recovery procedure exists only in the head of whoever set it up.
Misreads this invites
  • "StatefulSet means Kubernetes can run databases properly." It provides identity, storage binding and ordering. Everything specific to your engine — election, replication, backup, upgrade — is still yours or an operator's (Why Stateful Workloads Are Harder).
  • "All databases should run on Kubernetes now." Nothing here argues that. For most teams a managed database is the better default, and a StatefulSet is what you use when you have a specific reason not to use one (Managed vs Self-Hosted on the Cloud side).
  • "Ordinal 0 is the leader." Ordinals are lifecycle positions. Leadership is decided by the data system, and assuming otherwise produces confident, wrong runbooks.
  • "Scaling down cleans up." Claims are retained deliberately. Scaling down leaves disks and, if you scale back up, stale data.
  • "Use a StatefulSet for anything with a volume." If replicas do not need distinct identities and distinct data, a Deployment with a shared claim or a managed service is simpler (Deployments: Declaring What Should Be Running).

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked
  • Deleting a pod returns it with the same name, the same volume and its data intact, without a full resync.
  • A rolling update completes one member at a time, with cluster health checked between members and no unplanned failover.
  • A pod disruption budget demonstrably blocks a node drain that would break quorum.
  • A restore into a new StatefulSet has been performed and timed (Restore Drills).
How you get back
  • The pod template rolls back like any other workload, in reverse ordinal order, one member at a time.
  • The data does not roll back. If the new version wrote in a new on-disk or wire format, reverting the image can leave you with data the old version cannot read (A Migration and a Deploy Are One Event).
  • The partition field is the real safety mechanism: upgrade the highest ordinal only, verify it, and roll the rest forward or back before the change is fleet-wide (Canary: One Percent, Then Five, Then Watch).
What to automate, and what stays human
  • Automate through an operator for your engine, so failover, backup and upgrade procedures live in code that has been used by other people (How to Automate Something).
  • Automate backup and restore verification independently of the operator, so a bug in it is not also a bug in your recovery.
  • Do not automate scale-down. Removing a member is a data decision and deserves a human (The Automation Trap).
What this costs
  • You get one platform, one deployment model and one set of tooling for everything, at the cost of owning data operations that a managed service would have owned.
  • Ordered, one-at-a-time rollouts are safe and slow: a large cluster takes a long time to upgrade, and the window where versions are mixed is correspondingly long.
  • Per-replica volumes give correct identity-to-data binding and pin each member to a zone, reducing scheduling flexibility exactly where you would most like it.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

  • KUBERNETES-SPECIFICOrdinal names, per-replica claim templates and ordered rollout are Kubernetes' answer. Outside a cluster the same needs are met by named instances with attached disks and a runbook for the order to touch them, or by a managed service where the ordering is the provider's problem and invisible to you.
  • DATABASE-SPECIFICWhether an ordered rolling restart is safe depends on the engine: a quorum system may tolerate one member down and not two; a leader-follower system needs the leader handled last or stepped down first. The controller enforces order, not safety.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendread-replicas
Cloudstateful-workloadsmanaged-vs-self-hosted-dbpersistent-data-and-containers
Databasereplicationcrash-recovery
Domains that do not exist yet
  • — Distributed Systems — why an odd number of members and a majority quorum is the usual shape.