Storage Lifecycle
Hot to warm to cold to archive to deleted, driven by how the data is actually read — and the retrieval cost that makes archive a trap for anything you might read again.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which of your data has not been read in a year, and what would it cost you — in money you cannot see and in latency you have not measured — to have put it somewhere cheaper?
Three consumers with incompatible needs share one dataset. Dashboards read the last few weeks constantly and want them instantly. Analysts reach back a year occasionally and will wait. Auditors and regulators read nothing at all until they read everything, at which point the answer "we archived it" must still resolve to bytes. A single storage decision has to serve all three, which is why the decision is per-partition rather than per-table (Who Actually Consumes This Data).
The unit is one partition of one dataset at one age. Not a table: a table spans years and the year-old partitions of it have completely different access characteristics from this week's. Lifecycle policy that operates on tables is too coarse to be useful, and lifecycle policy that operates on individual objects is too fine to be manageable — the partition is the unit where the access pattern is homogeneous and the operation is cheap (Partitioning).
Keep everything in the default tier forever. Storage is the cheapest component in the platform, retrieval is instant, no policy has to be written, no partition can be accidentally unavailable, and nobody has to predict which data will be needed. For the first years of a platform this is correct and the effort of tiering would be better spent elsewhere.
The dataset that grows fastest is the one nobody reads — raw event landing, debug payloads, intermediate outputs of a pipeline — and it grows monotonically because nothing in the system proposes deleting anything (The Raw Landing Zone).
- The dataset that grows fastest is the one nobody reads — raw event landing, debug payloads, intermediate outputs of a pipeline — and it grows monotonically because nothing in the system proposes deleting anything (The Raw Landing Zone).
- Failed-run output accumulates alongside successful output. Nobody cleans it because nobody can tell, from the outside, which partitions were published and which were abandoned mid-write (Atomic Publish).
- Table-format snapshots and time-travel versions are real retained bytes that do not appear in a table listing. A daily rewrite of a large table can retain a version per day without any obvious growth in the table itself (Open Table Formats).
- Someone finally writes a lifecycle rule, applies it by prefix, and it matches a prefix that a quarterly job depends on. The failure is discovered at quarter end, by an auditor.
- Data is moved to an archive tier to save on retained bytes, and a compliance request then requires reading a year of it. The retrieval is charged, is slow enough to matter, and the saving of the previous year is consumed in a single afternoon.
- A deletion request arrives for a customer whose rows are in an archived partition. Satisfying it means restoring the partition, rewriting it and re-archiving — three operations where the design assumed zero (Deletion Requests).
What is actually happening
- Storage tiers trade retained bytes against retrieval. Colder tiers hold the same bytes for a smaller ongoing charge and make getting them back slower, more expensive, or both. That is the entire mechanism, and every tier name in every provider is a point on that one trade (Object, Block and File).
- The trade is asymmetric in a way the names hide. Moving data colder is cheap and usually immediate; moving it back is the expensive direction, and on the coldest tiers it is not instant — there is a restore step with a duration, which means the data is *unavailable* rather than merely slower (Choosing a Storage Type).
- Because retrieval is charged separately from storage, a colder tier is only cheaper in expectation, and the expectation is over how often you read it. Data read once a quarter is usually cheaper cold; data read once a week is usually more expensive cold than it ever was hot, and no report will tell you that has happened (Fixed vs Variable Cost).
- Tier transitions are themselves operations. Moving a million small objects to a colder tier is a million requests, which on a small-file dataset can cost more than the storage it saves — the small-file pathology reappearing in a new place (File Size and the Small-Files Problem).
- Deletion is the only step that actually removes the ongoing charge, and it is the only irreversible one. Every other transition can be undone by paying; deletion cannot be undone at all, which is why it sits at the end of the chain and why it needs an owner rather than a policy alone (Data Retention).
- Analytical query engines mostly cannot read the coldest tiers directly. A partition in deep archive is not a slow partition, it is a partition that makes the query fail — so tiering changes the *set of answerable questions*, not just their cost (Query Engines).
The chain, and what each step actually costs you
Lifecycle is usually drawn as a one-way chain — hot, warm, cold, archive, deleted — and the picture is accurate as far as it goes. What the picture leaves out is that every step is a different trade, and that the direction of travel is cheap while the direction back is not.
The table below separates the three things that change as data gets colder: what the ongoing charge is proportional to, what it costs to read, and what an analytical engine can do with it. That third column is the one people forget. A partition in deep archive is not a partition that queries slowly; on most engines it is a partition that makes the query fail, which means tiering has silently changed the set of questions the dataset can answer.
The final row is qualitatively different from the others. Every transition above it is a bet you can buy your way out of, and deletion is the one that cannot be undone at any price. That asymmetry is why deletion needs a named owner and a grace period, while a transition to a colder tier can reasonably be automated.
- Transitions are charged per object, so a dataset of many small files can cost more to move than the move saves. Compact first (File Compaction).
- Table-format snapshots and time-travel versions are retained bytes that no table listing shows, and their retention window is a setting somebody defaulted (Open Table Formats).
- The raw layer is the cheapest to keep and the most expensive to lose, because it is what makes every downstream correction a re-run rather than an apology (Keeping Raw History: The Recovery Position and the Liability).
| Stage | Ongoing charge follows | Reading it costs | What an engine can do with it | The trap |
|---|---|---|---|---|
| Hot | Bytes held, at the highest rate | Nothing beyond the read itself | Query directly, at full speed | Everything stays here by default, and the default is never revisited |
| Warm | Bytes held, lower | A modest per-read charge | Query directly; the engine does not distinguish it | Suits data read occasionally, and "occasionally" is usually an assumption rather than a measurement |
| Cold | Bytes held, lower again | A meaningful per-read charge plus a minimum retention before the move pays off | Usually queryable, sometimes with added latency | Data moved here and then read weekly costs more than it ever did hot, and nothing reports the inversion |
| Archive | Bytes held, lowest | A restore operation with a duration, charged per object and per byte | Not directly queryable on most engines — it must be restored first | Only cheaper if you genuinely never read it. The honest test is whether you would delete it instead |
| Deleted | Nothing | Not available at any price | Nothing | The only irreversible step in the chain, and the only one that actually removes the cost |
Every provider offers a version of this chain and each draws the boundaries differently: the number of tiers, the minimum period data must sit in a tier before the transition is economic, whether retrieval is charged per byte or per request or both, how long a restore takes, and whether their own query services can read a given tier directly. Those specifics change and must be read from current documentation. The shape — colder means cheaper to hold and more expensive to read — has been stable for as long as tiered object storage has existed.
Choosing a tier is a bet on how often you will read it
The decision is not "how old is this data" and it is not "how big is it". It is a probability estimate: how likely is anyone to read this partition in the next period, and what happens to them if the answer takes a restore.
Age is the proxy everybody uses because it is free to compute and it is right most of the time. It is wrong in a specific and expensive way: periodic consumers. A dataset read hard at every quarter end looks completely abandoned for eleven weeks out of thirteen, and an age-based policy will archive it four times a year, each time just before it is needed.
The options below are ordered by how much confidence they require about future access. Notice that the first one — leave everything hot — is a legitimate answer and stays legitimate for longer than most teams assume. Tiering has an engineering cost, an incident risk and a governance surface, and paying those to manage a storage line that is not among your top drivers is a poor trade (What Actually Drives Data Platform Cost).
A dataset partition has aged past its period of heavy use. What do you do with it?
when The dataset is small relative to the platform, the access pattern is unknown or periodic, or storage is not among your dominant cost drivers. Also the right answer whenever the engineering time to build and operate a policy exceeds what it would save.
cost You pay the highest retained-bytes rate on data nobody reads, indefinitely, and the amount grows without a review point. Acceptable while storage is a small share of spend; a slow leak once it is not.
when Access is genuinely occasional and measured — not assumed — and the engine can still read it directly, so no consumer experiences a failure. The default for history older than the working window on a large dataset.
cost Every read now carries a retrieval charge, so a wrong estimate of read frequency inverts the saving silently. Minimum retention periods mean data moved and moved back can cost more than leaving it.
when You are confident it will not be read in the ordinary course of business, and a restore with a real duration is acceptable on the rare occasion it is. Typically retention-obligation data with no analytical consumer.
cost It leaves the queryable world. Queries spanning the boundary fail rather than slow, restores are charged per object and per byte, and deletion requests against it become a restore-rewrite-re-archive cycle.
when No retention obligation applies, no consumer has read it within a window longer than your longest-period consumer's interval, and the data is either reconstructible from the raw layer or genuinely valueless. The right answer for intermediate pipeline outputs far more often than it is used.
cost Irreversible. Every future question about that period is unanswerable, and you will not find out which question that was until someone asks it.
when The partition is derived and the inputs plus the transformation code are both retained. Common for intermediate layers and for aggregates that can be rebuilt on demand.
cost You trade retained bytes for compute at retrieval time, plus the risk that the transformation is no longer reproducible because it depends on a source that has since changed (Reprocessing vs Retrying).
What tiering actually moves, and what it does not
Teams that build a lifecycle policy usually expect it to be a significant cost win, and are surprised when it is not. The reason is that storage is a driver proportional to data volume, and data volume is the slowest-growing large quantity in a platform. Scans and repeated compute grow with consumers and schedules, which are organisational, and organisations grow faster than datasets.
That does not make lifecycle work pointless — it makes it a second-order lever with a first-order risk, and worth doing in that order. The weights below break the storage line into its own components so the ordering within it is visible, because within storage the surprises are real: the largest retained bytes on many platforms are not the tables anyone would name.
The bottom two rows are the ones worth acting on first, and neither is a tiering decision. Failed-run output and abandoned intermediates are pure deletion candidates: they have no consumer, no retention obligation and no reconstruction risk, and removing them is the rare cost change with no trade at all.
The largest retained volume and the one to touch last. It is what makes every downstream correction a re-run, so its retention is a recoverability decision rather than a cost one.
Reconstructible from raw by definition, and frequently retained forever because nobody wrote a policy. The shortest justified retention in the platform and the first place to look.
Grows with rewrite frequency rather than with data volume, so a small table rewritten nightly can retain more than a large one written once. Invisible in any table listing; moved by setting an explicit version-retention window.
Small relative to raw and the most expensive to lose in terms of consumer disruption. Almost never the right place to save storage, and almost always where people look first because these are the tables they can name.
Pure waste with no consumer and no obligation. Moved by cleaning up on failure and by publishing atomically so partial output never lands in the final location at all.
A table copied for an investigation and never removed, multiplied by every investigation. Moved by making copies expire by default rather than by asking people to remember.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights within the storage driver only, on a modelled platform. Two things in this ordering are consistently counter-intuitive: the reconstructible intermediates outrank the serving tables people worry about, and snapshot retention is a real line that appears in no listing.
How to build it
Most important first.
- Drive the policy from measured access, not from age alone. Age is a convenient proxy and it is wrong for exactly the datasets that matter: a table read heavily at quarter end looks abandoned for eleven weeks out of thirteen (The Data Catalog).
- Set the policy per layer, with a stated reason. Raw data is the cheapest thing to keep and the most valuable to have kept, because it is what makes reprocessing possible; intermediate outputs are usually reconstructible and should have the shortest retention in the platform (Raw, Staging, Curated: Layers by Purpose, Keeping Raw History: The Recovery Position and the Liability).
- Compact before you tier. Transitions and restores are charged per object, so a dataset of many small files should be consolidated first or the transition costs more than it saves (File Compaction).
- Keep anything an engine must query directly in a tier the engine can read. Treat deep archive as a separate system with a restore step, and say so in the dataset's documentation rather than letting a query discover it (Dataset Documentation).
- Write down what makes a partition eligible for deletion, and make the check reversible for a grace period: stop refreshing, mark deprecated, wait past the longest-period consumer, then delete (Impact Analysis).
- Handle time-travel and snapshot retention explicitly. They are the retained bytes nobody lists, and the default retention on a table format is a policy someone else chose for you (Open Table Formats).
- Separate the compliance question from the cost question. Data that must be retained for a legal period is not a candidate for deletion at any price, and data that must be *deleted* by a legal deadline is not a candidate for archive at any price (Data Governance).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Every tier guarantees durability of the bytes. That is the one promise that does not weaken as data gets colder, and it is what makes tiering a cost decision rather than a risk decision (Object Storage as Data Infrastructure).
- Colder tiers do not guarantee availability on the timescale a query expects. A restore has a duration, and during it the data exists and cannot be read — which is a different failure from "slow" and needs a different consumer conversation.
- Nothing guarantees that a lifecycle rule matched what you thought it matched. Prefix-based rules are string matching, and the dataset a rule accidentally covers is discovered by its absence (Data Lineage).
- Deletion under a lifecycle rule is not a guarantee of compliance. Deleting a partition does not delete the copy in a downstream mart, an extract in a notebook, or a training set built from it (Deletion Requests).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a read-recency and reachability sweep: for each dataset, the newest partition, the oldest partition still in a query-readable tier, and the last time any query touched a partition older than the tiering threshold. It catches a policy that has archived data people are still asking for.
- It misses low-frequency consumers whose interval exceeds the window of query history you retain — which is the exact failure the whole lesson warns about, so the retention of the query log has to be longer than the longest consumer interval or the check is decorative.
- It also misses correctness entirely. A perfectly tiered dataset with an accurate policy can be missing a month of rows, and no lifecycle check will notice, because lifecycle operates on partitions and not on their contents (Volume Anomalies).
- Tiering has no effect on freshness at the head of a dataset, because the data being written is always in the hottest tier. It changes the *latency of history*, which is a property nobody measures and everybody assumes.
- The consumer-facing statement is therefore two-part: how fresh the newest data is, and how far back the dataset can be read at query speed. The second half is routinely missing from dataset documentation, and it is what breaks a year-over-year comparison (The Freshness SLO).
- A restore from archive is a latency the consumer experiences as an outage. Whatever the duration, it is not a number the query planner knows about, so the query does not wait — it fails or returns a partial result depending on the engine.
- Archived partitions keep the schema they were written with. A schema change applied to the live table leaves the archive holding a different shape, and a reader spanning the boundary must reconcile two schemas — with the older one being the expensive side to fetch (Schema Evolution).
- A partition-key change strands the archive under the old layout. Migrating it means restoring, rewriting and re-archiving everything, which is usually not done — so the historical part of the dataset keeps the old physical layout indefinitely (Physical Data Layout).
- Meaning changes are worse in cold storage than anywhere else, because the only record of what a column meant in 2021 is now in a tier nobody browses casually. Documentation is the mitigation and it has to be written at the time (Semantic Changes).
- Everything before deletion is recoverable by paying: restore the partition, use it, and re-tier it. Budget for the restore rather than treating it as a failure, because a tier you never restore from is one you should have deleted.
- Deletion is the one irreversible step in this lesson and possibly in the module. Before it, take an inventory of readers that spans longer than your longest-period consumer, and prefer a deprecation period where the data still exists but the pipeline no longer maintains it (Impact Analysis).
- Object versioning protects against an accidental overwrite and does not protect against a lifecycle rule doing exactly what it was configured to do. Test lifecycle rules against a listing before enabling them, not after (Data Retention).
- If raw history is intact, most tiering mistakes on derived data cost a recompute rather than the data. That is the strongest argument for treating the raw layer's retention differently from everything else (Reprocessing vs Retrying).
What can go wrong
- A lifecycle rule matching a prefix broader than intended, deleting or archiving a dataset a low-frequency job depends on.
- Archive chosen for data that turns out to be read regularly, so retrieval charges exceed the storage saved and nothing surfaces the inversion.
- A million-object transition costing more in requests than the tier change saves in retained bytes (File Size and the Small-Files Problem).
- A query failing rather than slowing on archived partitions, which reads to the consumer as a broken dataset rather than as a policy.
- The mitigation failing: an access-driven policy built on query history that is retained for less time than the longest consumer interval, which reports quarterly datasets as unread.
- Compliance and cost policies applied by different teams to the same prefix, so data is deleted before a retention obligation expires or retained past a deletion deadline (Data Governance).
- "Archive is just cheaper storage." It is cheaper retained bytes plus a charged, slow retrieval. It is only cheaper overall if you genuinely never read it, and the honest test is whether you would be willing to delete it instead (Choosing a Storage Type).
- "Nobody has queried it in ninety days, so it is safe to archive." Ninety days does not cover a quarterly close, an annual audit or a year-over-year comparison. The look-back has to exceed your longest consumer interval, not a round number.
- "Deleting the raw layer saves the most, because it is the biggest." It is also the cheapest layer per byte and the only thing that makes every downstream mistake recoverable. Delete intermediates first (Keeping Raw History: The Recovery Position and the Liability).
- "The lifecycle rule handles retention compliance." A lifecycle rule deletes objects under a prefix. Compliance is about a subject's data wherever it went, including into marts, extracts and models (Deletion Requests).
- "Our table is small, so we have no storage problem." Check snapshots and time-travel versions before believing that. A frequently rewritten table can retain far more history than its current size suggests (Open Table Formats).
- Retention has two masters that point in opposite directions. Some data must be kept for a statutory period and cannot be deleted for cost reasons; some must be deleted by a deadline and cannot be archived to defer the work. A single lifecycle policy that does not distinguish them will violate one of them (Data Retention).
- Archived data is still in scope for deletion requests. If satisfying one means restoring, rewriting and re-archiving a partition, that cost and that latency must be part of the deletion process design rather than discovered during the first request (Deletion Requests).
- Classification should drive tiering, not only access control. Data classified as sensitive that is also rarely read is the strongest candidate for deletion rather than archive, because retaining it holds risk with no offsetting value (Data Classification, Data Minimization).
Operating it
- Bytes per dataset per tier, trended. The report that makes the growth of the unread layers visible before it is a problem (Cost per Service and the Attribution Problem).
- Last-read timestamp per partition age band, from query history retained longer than the longest-period consumer (The Data Catalog).
- Restore operations per month, with the dataset and the requester. A rising count is the signal that a tiering decision was wrong, and it is the only signal there is.
- Object count per dataset before a planned transition, because the transition is charged per object rather than per byte (File Size and the Small-Files Problem).
- Snapshot and time-travel bytes as a separate line from table bytes, since they are retained storage that no table listing shows (Open Table Formats).
- At 10x volume, tiering moves from an optimisation to a routine part of dataset design, and the layer with the shortest justified retention — intermediate outputs — becomes the first place to look (Raw, Staging, Curated: Layers by Purpose).
- At 10x partitions, the per-object cost of transitions starts to matter and compaction becomes a precondition for tiering rather than an unrelated maintenance job (File Compaction).
- At 10x datasets, the policy cannot be maintained by hand and has to be a property attached to the dataset at creation, defaulted by layer and overridable by its owner (Data Ownership).
- History length scales the archive question specifically. A platform with three months of history has no lifecycle problem; one with seven years has a lifecycle problem and a compliance problem that interact (Data Retention).
- Retained bytes multiplied by time held, which is the driver everybody names and which moves slowly, because it grows with the data rather than with the organisation.
- Retrieval and restore operations, charged when cold data is read, which is the driver that makes a colder tier a bet rather than a saving.
- Transition operations, charged per object, which is why file count decides whether a tier change is worth performing at all.
- The unmeasured cost of unavailability: a partition in deep archive is not slow, it is absent for the duration of a restore, and a query that needed it does not complete.
- Every tier transition trades an ongoing saving for a contingent charge and a latency you cannot predict. The trade is good when the read probability is genuinely low and terrible when it is merely assumed to be.
- Deletion is the only step that fully removes cost, and it is the only step that removes optionality permanently. A platform that deletes aggressively is cheap and cannot answer new questions about its past.
- An access-driven policy is more correct than an age-driven one and requires query-history infrastructure, retention of that history, and an owner. Age-based policy is free and wrong for exactly the periodic consumers that matter most.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe trade of retained bytes against retrieval latency and retrieval charge is physical and holds on every storage system, including on-premises tape. What differs is where each provider draws the tier boundaries and how long a restore takes.
- CLOUD-SPECIFICTier names, minimum retention periods before a transition pays off, restore durations and whether an engine can read a tier directly are all provider decisions, and they differ enough that a policy copied between providers will behave differently. The shape transfers; the thresholds do not.
- ORG-SPECIFICWhether deletion is available at all is a legal question before it is an engineering one, and the answer varies by jurisdiction, industry and contract. In regulated settings the lifecycle chain often ends at archive, which changes the economics of every step before it.
- FORMAT-SPECIFICTable formats retain snapshots and previous file versions for a configurable window, so retained bytes exceed current table size by an amount that depends on rewrite frequency. A plain directory of Parquet files has no such hidden retention, and a table under Iceberg, Delta or Hudi does.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the lifecycle rules themselves — how they are expressed as infrastructure code, reviewed before they run, and tested against a listing rather than against production.