Keeping Raw History: The Recovery Position and the Liability
An immutable copy of what arrived is what makes every downstream mistake fixable. It is also the most sensitive dataset the platform holds. Both are true and neither cancels the other.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What can you still fix a year from now, and what have you promised to be able to delete?
Two consumers whose needs point in opposite directions. The engineer fixing a six-month-old bug needs raw to exist, be complete and go back far enough. The privacy officer answering a deletion request needs raw to be enumerable, deletable, and no larger than necessary. Designing for one and discovering the other during an audit is the normal failure (Data Governance).
One delivered record, exactly as delivered, with the metadata that makes it locatable: what it came from, when it arrived, which run wrote it, and which offset or log position it corresponds to. That metadata is what turns raw from a pile of files into something you can reprocess a bounded range of, and it is nearly always added later at considerable cost (The Raw Landing Zone).
Keep everything forever. Storage is the cheapest resource in the platform, the retention question is unpleasant, and deferring it costs almost nothing today. Almost every platform starts here and most stay here until something forces the conversation.
A deletion request arrives with a legal deadline. The raw layer is immutable files with no row-level delete and no index of which files contain which subject, so satisfying it means rewriting an unknown number of files and nobody can commit to a date (Deletion Requests).
- A deletion request arrives with a legal deadline. The raw layer is immutable files with no row-level delete and no index of which files contain which subject, so satisfying it means rewriting an unknown number of files and nobody can commit to a date (Deletion Requests).
- A field reclassified as sensitive turns out to have been landing in raw for three years. The remediation is not a filter change; it is a discovery, deletion and proof exercise across every derived dataset (Data Classification).
- Raw exists but is incomplete. The connector was configured to mirror current state rather than append changes, so what looks like three years of history is a fresh snapshot with a long file list (Snapshot and Stream: the Bootstrap Problem).
- Raw exists and cannot be reprocessed. Files were written without a load partition or a source watermark, so "re-run March" means scanning everything and filtering, which is affordable the first time and not the second (Partitioning).
- Retention was set to a number that sounded prudent and is shorter than the age of the oldest bug anyone has found. The recovery story was believed for two years and was never true (Data Retention).
- Raw is queryable by everyone because it was created before anyone thought about access, and the most sensitive dataset in the company has the loosest grants in the warehouse (Data Access Control).
What is actually happening
- Everything downstream of raw is a *function*. Staging, curated, marts and dashboards can all be recomputed from an input that was never modified — which means every transformation bug, every wrong business rule and every schema misinterpretation is repairable as long as that input exists (Reprocessing vs Retrying).
- Raw is the one thing that is not a function of anything. It is the only layer whose loss cannot be remedied by compute, and that asymmetry is the entire argument for keeping it (Raw, Staging, Curated: Layers by Purpose).
- The same property that makes it recoverable makes it dangerous. Raw is complete, source-shaped and unfiltered: it contains the fields nobody chose to keep, the values that were later corrected, and the records of subjects who have since asked to be forgotten (PII in Pipelines).
- Deletion is a physical problem, not a policy one. On plain immutable files, removing one subject means rewriting every file that contains them; on an open table format with row-level deletes, it is a bounded operation. This choice is made at platform-design time and determines whether a compliance commitment is achievable (Open Table Formats).
- Retention is a recovery-window decision. The correct framing is not "how long must we keep this" but "how old is the oldest mistake we would want to fix, and what would it cost us not to". Framed as storage cost, retention is always cut; framed as recovery window, it is argued honestly (Backfills).
- The two obligations can be reconciled, mostly, by separating them: minimise what enters raw, partition it so deletion is bounded, tier it so old history is cheap, and set a retention window per classification rather than per platform (Data Minimization).
Everything downstream is a function; raw is not
The lineage below is the technical content of this lesson in one table. Read the couldCorrupt column downward and notice that every entry describes something a re-run repairs — a wrong join, a wrong rule, a wrong grain. Then read the first row, where the failure is not corruption but absence, and no amount of compute repairs it.
That asymmetry is why raw retention is not comparable to any other storage decision in the platform. A deleted mart costs a rebuild. A deleted staging model costs a rebuild. A deleted raw partition costs the ability to answer a question, permanently, and the question usually has not been asked yet.
It is also why the argument is so easy to lose. During a cost review, raw is the largest storage line and the one nobody queries, which makes it look like waste by every metric on the slide. The counter-argument is not about storage at all: it is that the platform is selling its ability to restate history, and that should be a decision with a name on it.
- Source system
holds Current state. Mutable, and under someone else's change control.
could corrupt Nothing this domain owns — but it overwrites the past continuously, which is why re-extraction is not a general recovery path (Source of Truth).
↑ reads from - Raw landing
holds Every delivered record, unmodified, with load partition and source position.
could corrupt Only by loss: an expired retention window, an overwrite-mode connector, or a period the connector never delivered. Nothing downstream can recover this.
↑ reads from - Staging models
holds One row per entity, deduplicated and typed.
could corrupt Choosing latest by arrival rather than commit position; dropping deletes. Recovered by re-running from raw.
↑ reads from - Curated models
holds The analytical grain with business rules applied.
could corrupt Fan-out joins, wrong grain, a business rule that was wrong for six months. Recovered by re-running from staging or raw.
↑ reads from - Marts and extracts
holds Pre-aggregated copies for specific read patterns.
could corrupt Drift from the parent model after an in-incident patch. Recovered by rebuilding from curated.
↑ reads from - Dashboards
holds Numbers and BI-layer filters.
could corrupt A calculated field nothing upstream can see. Recovered by editing the dashboard — and only if someone finds it (Where the Transformation Actually Runs).
Five of six rows are recoverable from the row above. The one that is not is the one that costs the least to keep and is the first to be proposed for deletion.
The retroactive question raw makes answerable
The clearest demonstration of what raw buys is a schema change. A source adds a field. Under a pipeline that discards what it does not currently use, the field begins existing on the day someone writes a model for it, and the history before that is empty — permanently, because the source has overwritten it.
Under a pipeline that lands raw untouched, the field has been arriving all along. Writing a model for it produces a full historical series on the first run, and the difference between those two outcomes is one line of design made years earlier.
The schema diff below shows the change and who it reaches. Note the silent column: the consumers that break loudly are the fortunate ones. The genuinely expensive impact is the analysis that quietly cannot be done, and nobody files a ticket for a question they did not know was answerable.
- order_id
- customer_id
- amount_minor
- currency
- status
- occurred_at
- order_id
- customer_id
- amount_minor
- currency
- status
- occurred_at
- acquisition_channel
change A nullable string field is added upstream. No existing field is renamed, retyped or removed, and no consumer is notified.
| Consumer | Effect | How it shows up |
|---|---|---|
| Raw layer storing the full payload | The field lands from the first event that carries it. When a model wants it next year, the full history since today is already present. | Silently — no error, wrong result |
| Raw layer with an enforced strict schema | Records are rejected or the field is stripped. The history is not captured and the loss looks like the field simply not existing yet. | Silently — no error, wrong result |
| An ETL extract with an explicit column list | Nothing arrives. The pipeline succeeds, the destination is unchanged, and the field begins existing only when someone edits the extract. | Silently — no error, wrong result |
| A staging model using `SELECT *` | The column propagates downstream unannounced, changing the shape of every dependent model and any consumer relying on column order. | Silently — no error, wrong result |
| A curated model with an explicit column list | No change, which is correct behaviour: the curated contract only changes deliberately (Data Contracts). | Loudly — it raises |
| A marketing analyst next quarter | Can produce channel attribution across all of history, or across none of it, depending entirely on which of the first three rows describes the platform. | Silently — no error, wrong result |
The same dataset, seen from the privacy desk
Everything above is an argument for keeping more, for longer. The privacy view of the identical dataset is an argument for keeping less, for shorter, and it is not a weaker argument — it is a legally binding one in most jurisdictions where personal data is involved.
The two are reconcilable, but only through design rather than compromise: minimise what lands, partition so that deletion is bounded, classify at the field level so policies propagate, tier so that a long window is affordable, and set retention per classification instead of per platform. A single global retention number satisfies neither side.
The checks below are the ones that make the reconciliation verifiable. Each has a blind spot stated, because a governance check whose limits are unstated is exactly the kind of false confidence that turns a routine request into an incident.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Reproduce a closed period from raw and diff against published output. | The recovery path works end to end: raw is complete enough and the transformations are deterministic. | An incomplete raw layer, a non-deterministic transformation, a missing load partition, a broken reprocessing script. | Anything about periods older than retention; and it passes on a pipeline that is reproducibly wrong, because determinism is not correctness. |
| Reconcile raw row counts against the source for closed periods. | What the source sent is what raw holds. | Connector gaps, silently dropped tables, a snapshot restart that skipped an outage window. | Duplicates that offset losses; any period the source itself considers open; and value-level corruption, which preserves counts perfectly. |
| Assert forbidden fields are absent from every raw dataset. | Data minimisation at the landing boundary is actually being enforced. | A filter that was reordered, disabled, or never applied to a newly added source. | Sensitive values inside a free-text field that no column-level check inspects; and fields that are fine today and reclassified tomorrow. |
| Locate a known subject across raw partitions within a bounded time. | A deletion request can be executed on a schedule you can commit to. | A layout where a subject's records are spread across every file; a missing locator index; a format with no row-level delete. | Derived datasets and extracts that contain the subject without the raw identifier — a hashed key, a joined attribute, a copy in a notebook (Data Lineage). |
| Compare oldest retained partition against the stated retention policy, per source. | The recovery window you believe you have is the one you have. | Silent drift: lifecycle rules applied to one prefix and not another, or a policy changed without the storage configuration following. | Whether the stated window is the right one — which is a judgement about the age of your worst undiscovered bug, and no check can supply it. |
Two of these serve recovery and three serve governance, and a platform that runs only one group will be surprised by the other. They belong in the same suite, owned by the same people, reviewed together.
Retention as a recovery window with a price
The most useful reframing in this lesson is to stop asking "how long should we keep raw" and start asking "how old is the oldest mistake we would want to be able to fix, and what does not fixing it cost". Asked the first way, the answer is set by whoever owns the storage line. Asked the second way, it is set by whoever owns the metric.
The options below are the real ones. Note that they are not ordered from worse to better: a short window on a low-value, high-volume telemetry source is correct, and an indefinite window on a regulated financial ledger is also correct. What is never correct is one number applied to every dataset because it was easier than classifying them.
The cost shape underneath shows why the argument keeps being lost. Retained raw storage is a visible, growing line with no obvious consumer, and the thing it buys is invisible until the quarter you need it. Making the recovery window an explicit, named commitment — with an owner — is the only thing that reliably survives a cost review (Cost Attribution).
Has no line item and no owner, which is why it loses arguments to lines that do. This is the thing retention is purchasing.
Scales with retention and with how badly raw is partitioned. The cost of a long window paid on the governance side.
Paid only when you actually restate, and dominated by whether the layout lets you read one range instead of everything.
Small, growing, and the line every cost review points at first.
The cheapest way to keep a long window, at the price of slow retrieval — which is the correct trade for history nobody reads on a normal day.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights to establish an ordering, not measurements. The teaching is the first row: the dominant cost of a retention decision is the one that never appears on a bill, and every framing that omits it produces the same answer regardless of the data.
What is the oldest mistake you would want to be able to fix here, and what obligation runs against the same data?
when High-volume, low-value records where the aggregate is the product: verbose telemetry, per-frame sensor data, debug streams.
cost Any bug older than the window is unfixable. Acceptable precisely because nobody restates telemetry, and dangerous the moment someone tries.
when The common case: business data where restatement is plausible, deletion obligations exist, and volume makes an indefinite hot copy expensive.
cost Historical reprocessing becomes slow rather than impossible, and deletion must reach the archive tier too — which is the part most often forgotten.
when Restatement risk is high but the payload contains personal data: land the source with sensitive fields dropped or tokenised at the boundary.
cost The dropped fields are permanently unavailable for future questions. This is the ETL: Transform Before the Data Lands trade applied to the raw layer, and it is usually the right one.
when Regulated data with a statutory retention requirement, or a ledger whose history is the product.
cost Every governance obligation runs for the full period: classification, access review, deletion tooling, encryption key rotation. Retention this long is an operational commitment, not a storage setting.
when One source carries fields with different obligations — most operational databases.
cost More configuration and a policy that has to be recorded per field rather than per table, which needs field-level classification to exist first.
How to build it
Most important first.
- Land raw before you transform, under either ordering. This is the single highest-value structural decision in the module and it is available whether you call the pipeline ETL or ELT (ETL vs ELT: Choosing by Constraint, Not by Fashion).
- Write raw immutably, partitioned by load time, with a per-record source position — an offset, a log sequence number, a page cursor. Without that position, targeted reprocessing is not possible and "re-run March" becomes "re-run everything" (The High-Water Mark).
- Restrict raw to the pipeline. It is the platform's most sensitive dataset and its grain is wrong for every business question, so consumer access to it is both a governance risk and a correctness risk.
- Filter classified fields *before* they land, even under ELT. Loading everything is a default, not a principle, and there are fields whose presence in raw is itself the incident.
- Choose a storage layer that supports row-level deletion before you need it, then tier what it holds: recent raw hot for reprocessing, older raw archived, oldest aggregated or dropped, with retention set per classification rather than per platform. Retrofitting deletability onto a lake of plain immutable files is a migration, not a configuration change (Storage Lifecycle).
- Exercise the recovery periodically. Reprocess a closed historical period from raw on a schedule and compare it to what is published. A recovery capability that has never been used is a belief about a system, not a property of it (Restore Testing).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Raw guarantees that what was delivered is retained unmodified, and therefore that every downstream dataset is reconstructable from it. That is the strongest recovery guarantee a platform can offer.
- It guarantees nothing about completeness relative to the source. A connector gap, a missed DDL change or a snapshot restart is invisible from inside raw — it looks exactly like a period when nothing happened (CDC Failure Modes and the Retention Deadline).
- It does not guarantee reprocessability by itself. That requires load partitioning, source positions and deterministic transformations; raw without those is an archive rather than a recovery position (Determinism: Same Input, Same Output?).
- Immutability guarantees you cannot corrupt history and guarantees you cannot delete from it either. Those are the same property viewed from two desks.
- Retention guarantees a window, not a capability. A window longer than the oldest bug is a recovery story; shorter, and it is a storage line item.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that proves raw is real is a periodic reproduction: pick a closed historical period, rebuild the curated tables from raw into a scratch location, and compare row counts and summed measures against what is published. A match proves the recovery path works end to end (Reconciliation).
- A second check reconciles raw against the source for closed periods, because raw can only prove what was delivered and completeness is a claim about what was *sent*.
- They miss two things. A raw layer that is complete and reprocessable but whose retention is shorter than your oldest undiscovered bug passes both checks and will still fail you. And neither check says anything about whether raw contains data you should not be holding.
- Raw is the freshest layer in the platform, which is exactly why consumers find it and use it. Its freshness is a hazard rather than a feature, because its grain is the source's and every metric written against it is wrong in a plausible way (Grain: What Does One Row Represent?).
- Raw retention has no effect on freshness whatsoever. Conflating the two — cutting retention to "keep the platform fast" — is a category error that appears in real cost reviews.
- The freshness that matters for recovery is different: how quickly can you reprocess a historical range once you have decided to. That is a function of partitioning and compute, and it is the number to know before an incident rather than during one (Partition Pruning).
- Raw absorbs schema change by design: a new field lands whether or not any model uses it. This is the property that lets you answer a question retroactively, across history, about a field nobody was interested in at the time (Schema Evolution).
- That only works if raw is stored in a form that tolerates unknown fields. A raw layer with a strict enforced schema rejects the new field and the retroactive answer is unavailable (Forward Compatibility).
- A reclassification is the evolution raw handles worst. Fields that were fine to hold and are now not have been accumulating for years, and the remediation is proportional to retention rather than to the change.
- Meaning changes leave raw untouched and invalidate every model built on it. That is the good case: raw preserved the evidence, so the restatement is possible (Semantic Changes).
- This lesson *is* the recovery field of every other lesson in the domain. With raw intact, complete and partitioned, the repair for almost any downstream mistake is: fix the code, reprocess a bounded range, validate, publish (Planning a Backfill).
- Reprocess into a location consumers are not reading and swap atomically. Rebuilding in place is how a correction for last March becomes an outage this morning (Atomic Publish, What Backfills Break).
- Reprocessing is only correct if the transformation is deterministic. Any dependence on
now(), on a mutable dimension's current state, or on non-deterministic ordering makes the re-run produce a different answer than the original — which is not a repair. - Validate before publishing: compare the rebuilt range against the published one, expect a difference in exactly the dimension you fixed, and investigate any difference you did not predict (Validating a Backfill Before You Publish).
- Where raw has aged out, the fallback is the event log if one exists and its retention is longer, and an apology if it is not (Replay from the Log).
What can go wrong
- Raw that is a mirror of current state rather than an append-only history, so the recovery story was never true.
- Raw without load partitioning or source positions, so reprocessing a range means scanning everything.
- Retention shorter than the age of the oldest bug the platform has ever found — often set that way by a cost review where nobody stated which recovery window was being sold (Cost vs Freshness).
- The mitigation failing: a table format chosen for deletability, with the deletion tooling never tested, so the first real request is also the first execution of an untested path.
- Raw readable by every analyst, so the platform's most sensitive dataset has its loosest grants and metrics get written at the wrong grain.
- A transformation that reads the clock, making every re-run a different computation and every "reproduction" a new result.
- "Storage is cheap, so keep everything." Storage is cheap; obligations are not. Every retained record carries retention, deletion, access and classification duties that are paid in engineering and legal time.
- "We have raw, so we can fix anything." Only if raw is complete, partitioned, deterministic to reprocess, and older than the bug. Four conditions, each of which fails quietly.
- "Raw is a backup." A backup is restored after a disaster. Raw is read on an ordinary Tuesday to fix an ordinary bug, and it needs the availability and access characteristics of a working dataset (Backup Strategy is the different thing).
- "Deleting from raw breaks reproducibility, so we cannot do it." It changes reproducibility for the affected subjects, which is the intended outcome of a deletion request. Design for it rather than arguing against it.
- "Retention is a cost decision." It is a recovery-window decision with a cost. Presenting it as the former guarantees the window gets cut by someone who was never told what it was for.
- Raw is the highest-risk dataset in the platform and should have the narrowest grants. Most platforms have this exactly backwards because raw was created before anyone was thinking about access.
- Data minimisation applies at the landing boundary. Deciding not to land a field is prevention; deleting it afterwards is remediation, and the two differ by orders of effort.
- Deletion requests must reach raw and every derivative. The layered structure is what makes that enumerable; the storage format is what makes it possible; the partition layout is what makes it affordable.
- Retention should be set per classification, not per platform. A single number applied to everything is either too long for personal data or too short for the recovery window, and usually both at once.
- Every raw dataset needs a recorded classification at the field level, because that record is what lets masking, row and column policies, and deletion tooling apply automatically to everything derived from it (Row and Column Security).
Operating it
- Raw completeness per source per period, reconciled against the source for closed periods. This is the number that decides whether the recovery position exists.
- The age of the oldest retained raw partition per source, published next to the stated retention policy. Drift between the two is common and always surprising.
- Reads of raw by non-pipeline principals, from the query log. Any is a finding.
- Time to reprocess a standard historical range, measured by actually doing it on a schedule rather than estimated.
- Presence of a subject-locator index — which partitions contain which subject identifiers — because without it every deletion request is a full scan.
- At 10x volume, retention becomes a real conversation rather than a rounding line, and tiering is the answer that preserves the recovery window without preserving the price.
- At 100x, deletion feasibility dominates the design: a raw layer that cannot delete rows without rewriting large fractions of itself is a compliance liability that grows with success.
- At high source count, the per-source retention decision has to be recorded rather than remembered, and the classification of each source has to be recorded with it (Metadata: Technical, Operational and Business).
- At high subject count, the locator index stops being optional. Finding a subject by scanning is affordable at small scale and is the reason deletion requests take weeks at large scale.
- Retained raw storage grows monotonically and is usually the smallest meaningful line in the platform. It is also the line most often proposed for cutting, because it is the only one that grows without anyone doing anything.
- Tiering is the honest lever: recent partitions in the storage class that supports fast reprocessing, older partitions archived. Archive tiers make reprocessing slow rather than impossible, which is the correct trade for old history.
- Deletion costs compute proportional to how badly raw is partitioned. A layout where a subject's records are spread across every file makes each request a full rewrite (Physical Data Layout).
- The cost that is not on the bill is the one that matters: an unfixable year of a metric. That has no line item and it is the thing retention is actually buying.
- Keeping raw buys the ability to fix anything and costs a permanent, complete, source-shaped copy of data that someone will eventually have to account for. There is no version of this trade where you get both.
- Minimising what enters raw reduces the liability and forecloses future questions, exactly as a transform-before-load filter does. It is the same trade at a different point in the pipeline (ETL: Transform Before the Data Lands).
- Long retention buys a wide recovery window and lengthens the tail of every deletion and reclassification obligation. Short retention does the reverse, and the platform finds out which was right only when something goes wrong.
- Tiering old raw to archive storage keeps the window at lower cost and makes a historical reprocess slow. For history older than a year that is almost always the right choice, and it should be a stated property rather than a discovered one.
Raw retention lab
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Discovered after | Scenario | Can you rebuild? |
|---|---|---|
| 1d | A transformation that was deployed and reverted the same day. | pass |
| 7d | A modelling bug found by the weekly report — the most common detection interval there is. | pass |
| 31d | A definition change discovered at month-end close, which is when finance looks properly. | FAIL |
| 92d | A quarter that has to be restated, usually because someone external asked a question. | FAIL |
| 400d | A year-on-year comparison that reveals a bug present the whole time. | FAIL |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe asymmetry that everything downstream of raw is recomputable and raw is not holds in every data platform regardless of technology, and it is the reason the same advice appears in batch, streaming and lakehouse contexts that otherwise share nothing.
- FORMAT-SPECIFICWhether deletion from raw is bounded depends on the storage format: an open table format with row-level deletes turns a subject deletion into a metadata operation plus a small rewrite, while plain immutable files require rewriting every file containing the subject and re-pointing anything that referenced them.
- ORG-SPECIFICThe right retention window is a negotiation between engineering's recovery needs and legal's minimisation obligations, and the answer differs by industry and jurisdiction — a regulated financial dataset and a product clickstream can justify opposite conclusions from the same reasoning.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why replaying from a log is bounded by retention rather than by storage, and why "we can always replay" is a claim about a configured window rather than about a durable log.
- — DevOps / Production Engineering owns the disaster-recovery vocabulary this lesson borrows — recovery point and recovery time objectives — and the observation that a restore path which has never been exercised is an assumption rather than a capability.