Volumes: Storage With a Lifecycle
A container filesystem dies with the container. A volume is a way of saying which data outlives what — the pod, the node, or the cluster — and each answer has different failure modes.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What happens to data a container writes, and how do I choose how long it should survive?
Container filesystems are ephemeral by design, which is what makes containers reproducible. Real workloads still write things — caches, uploads, database files — and each needs a different lifetime.
Attach a persistent volume whenever a workload writes anything. It is one field, it makes data survive restarts, and it removes a class of surprises.
Attaching persistent storage constrains scheduling. A volume that can only be attached to one node ties the pod to that node, so a node failure means waiting for detach rather than rescheduling instantly (The Scheduler, and Why a Pod Is Pending).
- Attaching persistent storage constrains scheduling. A volume that can only be attached to one node ties the pod to that node, so a node failure means waiting for detach rather than rescheduling instantly (The Scheduler, and Why a Pod Is Pending).
- It turns a stateless workload into a stateful one operationally: now it has a backup requirement, a restore path and a capacity trend, none of which the manifest mentions (Backup Operations).
- Multiple replicas sharing one volume usually cannot. Most block storage supports a single writer, so a Deployment scaled to three with one volume produces two pods stuck waiting to attach.
- Persistent storage that outlives the workload accumulates. Orphaned claims keep provisioning real, billed disks long after the thing that used them is gone (Idle Capacity).
- It hides the real question. Data that must survive usually needs backups, replication and a restore drill — none of which a volume provides (Restore Drills).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A container's writable layer is destroyed when the container is removed. A volume is declared at pod level and mounted into containers, so its lifetime is a property of the volume type rather than of the container.
emptyDirlives as long as the pod on its node. Container restarts keep it; pod deletion or rescheduling destroys it. It is the right choice for scratch space and shared space between containers in a pod (Pods: The Unit That Gets Scheduled).- A PersistentVolumeClaim is a request for storage that outlives the pod. A provisioner satisfies it by creating a real disk, and the claim is bound to that volume until it is deleted.
- A StorageClass decides what kind of disk gets created and, critically, when:
volumeBindingMode: WaitForFirstConsumerdelays provisioning until the pod is scheduled, so the disk is created in a zone the pod can actually run in. - Access mode is a real constraint, not a hint. Typical block storage is single-node read-write; shared filesystems support many writers with different performance and consistency behaviour (File Storage on the Cloud side).
- The claim's reclaim policy decides what happens to the disk when the claim goes away — delete it, or keep it. That single field is the difference between "we cleaned up" and "we deleted the data" (Destructive Migrations is the same class of mistake in another layer).
Four lifetimes, four uses
Choosing storage is choosing a lifetime. Read the second column as the event that destroys the data — that is the question the choice actually answers.
| Kind | Destroyed by | Use it for | Watch out for |
|---|---|---|---|
| Container filesystem | Container restart | Nothing you want to keep | Logs written to disk vanish on restart |
emptyDir | Pod deletion or reschedule | Scratch space, sharing between containers in a pod | Node drains during upgrades destroy it silently |
| PersistentVolumeClaim | Claim deletion (if reclaim policy deletes) | Data that must outlive the pod | Pins the pod to a zone; single writer for most block storage |
| Shared filesystem claim | Claim deletion | Many pods reading and writing the same tree | Different performance and consistency behaviour from block storage |
| Managed service outside the cluster | A deliberate action in that service | Databases, queues, object storage | A separate access model, a separate bill, a network hop |
A claim, and the field that decides whether delete means delete
The manifest is short. Two of its fields — the storage class and the access mode — decide where the pod can run, and a field on the class decides whether deleting the claim destroys the data.
1apiVersion: storage.k8s.io/v12kind: StorageClass3metadata:4 name: standard-zonal5provisioner: example.csi.driver.io6reclaimPolicy: Retain # Delete would destroy the disk with the claim7volumeBindingMode: WaitForFirstConsumer # provision in the zone the pod lands in8allowVolumeExpansion: true9---10apiVersion: v111kind: PersistentVolumeClaim12metadata:13 name: checkout-data14spec:15 storageClassName: standard-zonal16 accessModes:17 - ReadWriteOnce # one node may mount it read-write18 resources:19 requests:20 storage: 50GiReadWriteOnce is why a Deployment with three replicas and one claim leaves two pods Pending. WaitForFirstConsumer avoids the classic multi-zone failure where the disk exists in one zone and the only schedulable node is in another. Retain means a deleted claim leaves the disk — and the manual cleanup — behind, which is the safer default for anything you would miss.
Storage failures and their signatures
Storage problems in a cluster rarely announce themselves as storage problems. They present as pods that will not start, replicas that will not scale, or latency nobody can explain.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Pod Pending with a volume | Never scheduled; no application logs exist | Disk is in a zone with no schedulable node, or still attached elsewhere | Read the pod events; use WaitForFirstConsumer binding to prevent the zone case |
| Second replica never starts | One pod Running, others stuck | ReadWriteOnce allows a single node to mount it | Either one replica, or per-replica volumes via a StatefulSet (StatefulSets: Identity, Storage and Order) |
| Node drained for upgrade | Cached or uploaded data gone | emptyDir used for something that mattered | Move it to a claim, or make the data reconstructible |
| Volume full | Writes failing; pod still reported healthy | No capacity signal, and readiness does not test writes (Probes: Readiness, Liveness and Startup) | Alert on free space with lead time; expand if the class allows it |
| Claim deleted during cleanup | Data gone, immediately and permanently | Delete reclaim policy on the class | Restore from snapshot; set Retain for anything you would miss (Partial and Logical Data Recovery) |
| Unexplained tail latency | p99 rises with no code change | Disk performance tied to class and size, and the workload outgrew it | Compare against the class's actual limits before profiling the application (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax) |
How to do it properly
Most important first.
- Start by asking how long the data must survive, and pick the weakest option that answers it. Most workloads need nothing; many need
emptyDir; a minority need persistence. - For data that genuinely matters, prefer a managed data service outside the cluster over a volume inside it, unless you have a reason and the expertise to run it yourself (Why Stateful Workloads Are Harder).
- Use
WaitForFirstConsumerbinding in multi-zone clusters, so storage and pod are not provisioned into different zones (Multi-Zone Deployment on the Cloud side). - Set the reclaim policy deliberately and know which one you have before you delete anything.
- Monitor volume capacity as a first-class signal. A full disk on a stateful workload is an outage that no amount of replica scaling fixes (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax).
- Back up what is on the volume, and restore it somewhere at a known cadence. A snapshot that has never been restored is a claim, not a capability (Restore Drills).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A volume mistake is contained to the workload that owns the data — except deletion with a delete reclaim policy, which is unbounded within that workload and has no undo but a restore.
What can go wrong
- Pod stuck in Pending on a volume that cannot be attached — most often because the disk is in another zone or still attached to a node that has not released it.
- Two replicas, one single-writer volume: one pod runs and the other waits forever, which reads as a scaling failure rather than a storage one.
emptyDirused for data that mattered, discovered when a node drain destroys it during a routine cluster upgrade.- A volume filling up. Writes fail in whatever way the application handles worst, and the platform reports the pod as perfectly healthy (Probes: Readiness, Liveness and Startup).
- A claim deleted with a delete reclaim policy, taking the underlying disk with it. This is instant, permanent, and looks like tidying up.
- Storage performance assumed from the class name. The provisioned disk's throughput and IOPS behaviour vary enormously by class and by size, and the difference shows up as unexplained tail latency (Tail Latency: Why p50 Being Fine Does Not Help).
- "A persistent volume means the data is safe." It means the disk outlives the pod. Safety is backups you have restored, plus replication (Backup Operations).
- "Volumes make a workload stateful." The data does. The volume is how you admit it (Why Stateful Workloads Are Harder).
- "Any pod can mount any volume." Access modes and zone placement constrain this heavily, and most block storage allows exactly one writer.
- "Deleting the claim just releases the storage." With a delete reclaim policy it destroys the disk and its contents, immediately.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The claim is Bound, and the pod using it is Running on a node in the same zone as the disk.
- A deliberately deleted pod comes back with its data intact — which is the actual test that persistence is configured the way you think.
- Free space on every persistent volume is monitored and trending, not checked when something breaks.
- A restore from backup into a fresh volume has been performed, with a recorded duration (RTO and RPO).
- Detaching a volume is reversible; deleting one is not. The reclaim policy is the field that decides which of those a
kubectl deleteperforms. - Rolling back an application version does not roll back what it wrote to the volume. Data written by the new version stays there, which is the same coupling problem as a schema migration (A Migration and a Deploy Are One Event).
- Restoring from a snapshot is the real rollback for data, and its recovery point is whenever the snapshot was taken (Partial and Logical Data Recovery).
- Automate provisioning through StorageClasses so no one creates disks by hand, and so zone and type decisions are made once (Infrastructure as Code).
- Automate snapshots, snapshot expiry, and a periodic restore into a scratch environment — the restore is the part that proves the rest (Restore Drills).
- Automate capacity alerting with enough lead time to act, rather than at the point the disk is already full.
- Keep deletion human and deliberate. A pipeline that deletes claims is a pipeline that can delete data (Destructive Changes: What a Rename Really Does).
- Persistent storage buys durability and costs scheduling flexibility: the pod is now tied to where its data is, which is exactly the property that makes stateless workloads easy to move.
- Managed data services outside the cluster remove the hardest operational work and add a network boundary, a separate access control model and a separate bill.
emptyDiris free and fast and gives you nothing on reschedule, which is fine right up until someone uses it for something that mattered.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICClaims, StorageClasses and binding modes are Kubernetes' indirection over provider storage. On plain VMs you attach a disk to an instance and the coupling is explicit and obvious; the Kubernetes model makes it dynamic and therefore easy to forget that a pod with a volume is pinned to wherever that volume can attach.
- CLOUD-SPECIFICWhich access modes exist, whether volumes can be expanded in place, whether they are zonal or regional, and what performance a class delivers are all provider and class decisions. Two clusters with identical manifests can behave differently for this reason.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.