Deletion Requests
A person asks to be erased from a platform built on immutable files, replayable logs and forty copies — and the backfill you run next week can bring them back.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A subject requests erasure. What actually has to happen, how do you prove it did, and what stops a replay from resurrecting them?
The privacy function that must attest the request was fulfilled, and the subject who is entitled to that. Also, immediately, the engineer who will run the next backfill — because deletion and reprocessing are the same mechanism operating in opposite directions, and a platform that treats them separately will undo one with the other (Backfills).
The unit is one subject across every copy, which is precisely the grain a data platform is worst at. Every mechanism in the platform operates on partitions, files, tables and batches; a deletion request operates on a person, whose rows are spread across all of them and are identified by a key that half the datasets no longer carry.
Delete the customer's row from the warehouse. DELETE FROM dim_customers WHERE customer_id = ..., confirm one row affected, and record the request as fulfilled. In an operational database this is genuinely most of the work, which is exactly why the analytical version is underestimated.
The raw landing zone holds the original payloads in immutable files. There is no row to delete — there is a Parquet file containing that row among a million others, and removing it means rewriting the file (The Raw Landing Zone).
- The raw landing zone holds the original payloads in immutable files. There is no row to delete — there is a Parquet file containing that row among a million others, and removing it means rewriting the file (The Raw Landing Zone).
- The event log holds every change the subject ever produced, retained by time or, on a compacted topic, retained as the latest value per key indefinitely (Retention and Replay).
- The dimension row is gone and the fact rows referencing its surrogate key remain, so the subject is still present as a behavioural trail with a dangling key — and the deletion has broken referential integrity rather than removing the person (Surrogate Keys).
- Deleting from an open table format writes a delete marker. Queries stop returning the row; the bytes remain in the data files until snapshot expiry and a rewrite, and anything reading the files directly still sees them (Open Table Formats).
- Backups and snapshots hold the pre-deletion state for the length of their own retention, and a restore silently reinstates the subject (Backup Strategy).
- The backfill resurrects them. Six weeks later a bug is fixed and March is reprocessed from raw. Raw was rewritten to remove the subject — or was not — and either way the reprocessing rebuilds the modelled tables from whatever raw currently holds. If raw was not cleaned, the person is back, in a table that was certified clean (What Backfills Break).
- A model trained on a dataset that included the subject encodes something about them in its parameters, and there is no row to remove (Embedding Pipelines).
- The subject appears in a support ticket, a pipeline log, an alert, a notebook checkpoint and a CSV on a laptop, none of which the deletion tooling has ever heard of (PII in Pipelines).
What is actually happening
- Erasure in an analytical platform is not a delete. It is a rewrite plus a suppression, and both halves are required. The rewrite removes the subject from every copy you can enumerate. The suppression prevents any future process from putting them back.
- The suppression list is the part people miss and the part that makes the whole thing work: a durable record of subjects whose data must not be materialised, consulted by every ingestion path and every reprocessing job. Without it, deletion is a point-in-time cleanup that any replay undoes (Replay from the Log).
- Immutable storage changes the cost model entirely. Removing one record from a columnar file means reading the file, filtering, writing a new one and expiring the old — so deletion cost is driven by how many *files* contain affected subjects rather than by how many rows are deleted (File Compaction).
- Table formats separate logical from physical deletion deliberately: a delete marker or an equality delete hides the record immediately and cheaply, and the bytes go away later, when snapshot expiry and compaction run. Both steps are required for erasure and only the first is automatic (Open Table Formats).
- Crypto-shredding sidesteps the rewrite for the cases where rewriting is impossible. Encrypt each subject's identifying data under a per-subject key; erasure is destruction of that key, and every copy — raw, backup, snapshot, an archive nobody can rewrite — becomes unreadable simultaneously (Data Masking, Tokenisation & Encryption).
- Deletion and anonymisation are different obligations with different costs. Removing a person's rows destroys history; replacing their identity with an unlinkable value preserves aggregate history and may satisfy the requirement. Which one applies is a legal determination, and the engineering differs enormously (Data Minimization).
- Proof is a first-class deliverable. "We ran the job" is not evidence; a per-request record of which datasets were scanned, which were affected, when the rewrite completed and when snapshots expired, is (Audit Logs for Privileged Actions).
Erasure is a rewrite plus a suppression
In an operational database, deleting a person is close to a single statement. In an analytical platform it is a distributed removal across immutable files, plus a durable statement that constrains everything the platform does in the future. Leaving out the second half is the mistake that defines this lesson.
The reason is that a data platform is a machine for rebuilding derived data from retained sources. That is its central virtue — it is what makes backfills, restatements and bug fixes possible at all — and it means any state you reach by deletion is a state the next reprocessing run will recompute from scratch (Backfills).
So erasure has two halves with different latencies. Suppression must be effective immediately and is cheap: a durable list, consulted by ingestion and by every reprocessing job. Rewrite is slow and expensive and can be batched. Every stage below has to promise something, and the stage that carries the guarantee is the suppression one.
- 1Authenticate and resolve
Verifies the requester and resolves them to every key the platform knows them by — user id, customer id, device identifiers, tokens.
guarantees That the deletion targets the right subject, and only that subject. Over-deletion is the unrecoverable error here.
fails by Matching on an unauthenticated identifier, or missing one of several keys the same person has across sources.
- 2Suppress
Writes the subject to a durable suppression list consulted by every ingestion path and every backfill.
guarantees That no process consulting the list will re-materialise the subject — the only guarantee that survives reprocessing.
fails by Being enforced by convention in each pipeline instead of in a shared path, so a pipeline added later never consults it.
- 3Scope
Derives the affected dataset list from column-level lineage, forward from the source columns.
guarantees Coverage of everything lineage can see. Nothing about copies outside the platform or downstream of a lineage gap.
fails by Being a hand-maintained list, which is stale the week a new dataset ships (Column-Level Lineage).
- 4Logical delete
Writes delete markers or removes rows so queries stop returning the subject.
guarantees Invisibility to queries through the governed engine, quickly. Not removal of bytes.
fails by Being mistaken for the end of the process, which is the most common way a request is closed prematurely.
- 5Physical removal
Rewrites affected files, expires snapshots, runs compaction.
guarantees That the bytes are gone from the datasets processed — after the expiry window, not at the time of the request.
fails by Snapshot expiry never being scheduled, so the delete markers hide data that is still fully present (File Compaction).
- 6Handle the unreachable
Destroys per-subject keys for archives and backups; schedules retraining or exclusion for models; records what cannot be removed.
guarantees For key destruction, simultaneous unreadability everywhere the key was required. For the rest, an honest record rather than a guarantee (Data Masking, Tokenisation & Encryption).
fails by Attesting completion while a trained model and a partner copy still contain the subject.
- 7Verify and attest
Queries every dataset in scope for the subject key and records the evidence per dataset.
guarantees Evidence of absence at a point in time, for the datasets checked.
fails by Recording the job's exit status as the evidence — the domain's own thesis, in its governance form.
Read the latencies: suppression must be immediate, rewrite can be batched, and physical removal completes on the schedule of a job most platforms have never checked is running.
The backfill that brings them back
Six weeks after an erasure is certified complete, someone finds a bug in a revenue transformation and reprocesses March from raw. The backfill is correct, it is reviewed, it validates cleanly, and it rebuilds fct_orders for March from the raw layer — which is where the subject's original rows live unless they were rewritten there too (Planning a Backfill).
Nothing in that sequence looks like a governance failure. The pipeline succeeded, the numbers reconcile, the tests pass. The only symptom is a row for a person who was erased, in a table that was attested clean, and the only way anyone finds it is by looking for it deliberately.
This is why the suppression list is the deliverable rather than the deletion script. Every reprocessing job, every replay from the log and every restore from backup must consult it as a filter, and that must be enforced in shared machinery rather than left to each pipeline's author to remember (Idempotent Data Pipelines).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A backfill reprocesses a historical period from raw. | The subject reappears in modelled tables that were verified clean. | Raw was not rewritten, or was rewritten but the job reads an older snapshot; either way no filter excluded the subject. | Apply the suppression list as a mandatory filter in the shared transformation path, and test it with a synthetic erased subject after every backfill. |
| A consumer replays from the event log to rebuild state. | Subject data returns to a downstream store that never held it after the deletion. | Log retention outlives the deletion, and replay is a supported operation that reads history directly (Replay from the Log). | Suppress at the consumer, not only at the producer; write tombstones where the topic is compacted; bound log retention deliberately. |
| A restore from backup after an incident. | Every subject erased since the backup was taken is reinstated, silently. | The restore procedure has no post-restore step, because the backup predates the deletion by design. | Make re-application of the suppression list a required, tested step in the restore runbook (Restore Testing). |
| A new source integration lands the same subject again. | A supposedly erased person appears in raw, then flows downstream normally. | Suppression was applied to the pipelines that existed at the time, and the new one does not consult it. | Enforce suppression in a shared ingestion library or gateway that every source must pass through, and alert on any path writing to raw that does not. |
| Snapshot expiry is not scheduled on a table-format dataset. | Queries show no rows; direct file reads and time-travel queries show everything. | Delete markers hide data; only expiry plus rewrite removes it (Open Table Formats). | Schedule and monitor expiry and compaction as first-class jobs, and verify deletion by inspecting files rather than by querying the table. |
| A model is retrained on an archived training set. | A model in production encodes a subject who was erased before it was trained. | Training sets are frozen snapshots by design, and freezing is what makes them reproducible. | Filter training-set construction through the suppression list, version training sets with their suppression state, and accept retraining rather than editing (Evaluation Data Pipelines). |
Proving it, and pricing it
A deletion request closes on evidence, not on a job status. The evidence is per dataset: this dataset was in scope, it was queried for the subject key, it returned nothing, and its snapshot expiry has run since the rewrite. Producing that automatically is what makes erasure operable at volume; producing it by hand is what makes teams quietly stop verifying (Reconciliation).
The cost side has one property worth internalising: deletion cost is driven by files touched, not rows deleted. A subject whose rows are scattered evenly across a large table touches nearly every file; a subject whose rows are concentrated — because the layout clusters by something correlated with them — touches few. Nobody chooses a partition or clustering key with erasure in mind, and it is one of the decisions that key silently makes (Clustering and Sort Order).
That is also the argument for batching. Fifty requests processed together rewrite each affected file once instead of fifty times, and the fulfilment delay is usually well inside any deadline — provided the physical phase is included in the calculation, which it frequently is not.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Subject key returns zero rows in every dataset named by lineage | Completeness of the rewrite across the enumerable copies. | A dataset missed by the scope, a rewrite that failed partway, a downstream mart rebuilt from stale input. | Datasets outside lineage — logs, tickets, extracts, notebooks — and any dataset where the subject is present without the key. |
| Snapshot expiry and compaction have run on each affected table since the rewrite | That physical removal followed logical deletion. | The extremely common case of delete markers with no expiry scheduled, where bytes remain fully readable. | Backups and replicas taken before the rewrite, which have their own independent lifetimes (Backups Are Sensitive Data Copies). |
| Synthetic erased subject is absent after a backfill of a period they were in | That suppression is enforced on the reprocessing path, not only at deletion time. | The resurrection failure — the one that no per-request check will ever find, because it happens weeks later. | Paths not exercised by the test: a replay from the log, a restore, or a new source added since the test was written. |
| Every writer to raw passes through the suppression-checking ingestion path | That suppression coverage is structural rather than per-pipeline. | A new source integration that lands subject data directly, bypassing shared machinery. | A pipeline that consults the list but caches it, so a recently suppressed subject is still admitted for the cache lifetime. |
| Time from request to verified absence, per request, against the deadline | That the process completes within its obligation, including the physical phase. | Batching intervals that ignore snapshot expiry latency, and requests stalled on one slow dataset. | Requests that were never created because the intake path failed — the check measures what entered the system, not what was asked for. |
The third row is the one that distinguishes a platform that has thought about this from one that has not. It is also the only check here that cannot be run at request time, because the failure it detects happens on a schedule nobody controls.
How to build it
Most important first.
- Build the suppression list first, before any deletion tooling. It is consulted by ingestion, by every backfill and by every replay, and it is the only thing that makes an erasure durable rather than momentary (Idempotent Data Pipelines).
- Derive the scope from lineage rather than from a maintained list. "Which datasets contain data about this subject" is a lineage query from the source columns forward, and a hand-written list is stale the week after it is written (Column-Level Lineage).
- Keep a subject key present in the datasets that need to be erasable, or you cannot find the rows. Deletion is the requirement that argues *against* dropping the join key from behavioural tables, and it must be traded against minimization deliberately (Surrogate Keys).
- Prefer per-subject encryption for the layers that cannot be rewritten — archives, backups, third-party copies — so that erasure there is a key destruction rather than a negotiation (Key Management and Encryption at Rest).
- Batch requests. Rewriting the same files once for fifty subjects instead of fifty times is the difference between a routine job and a permanent compaction workload, at the cost of a fulfilment delay you must be able to justify.
- Make the two-phase nature explicit in the process: logical delete now, physical removal when snapshot expiry and compaction complete, and the request is not closed until the second phase is verified (Atomic Publish is the analogous discipline on the write side).
- Decide the model and training-set policy in advance: retrain on a schedule, exclude suppressed subjects from every future training set, and record what cannot be removed as residual risk rather than claiming it was (Risk, Residual Risk and Honest Reporting).
- Test the whole thing with a synthetic subject: insert one, run the platform normally for a period, erase them, then run a backfill and go looking. This is the only test that catches resurrection (Data Tests).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A
DELETEagainst a warehouse table guarantees the rows are no longer returned by queries through that table. Whether the bytes are gone depends on the storage layer, and on a table format it usually means they are not (Open Table Formats). - A file rewrite plus snapshot expiry guarantees the bytes are removed from that dataset. It guarantees nothing about backups, snapshots, replicas or any copy taken before it ran (Backups Are Sensitive Data Copies).
- A suppression list guarantees that processes which consult it will not re-materialise the subject. Processes that do not consult it — a one-off script, a new pipeline, a restore — are outside the guarantee, which is why it must be enforced in a shared ingestion path rather than by convention.
- Crypto-shredding guarantees that data encrypted under the destroyed key is unreadable, everywhere, at once. It guarantees nothing about data that was decrypted and re-stored in clear somewhere downstream, which is the failure mode to design against.
- Nothing guarantees erasure from a trained model, a third-party system or a copy that left the platform. The honest posture is retraining, contractual obligations and recorded residual risk.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The resurrection test: erase a synthetic subject, then run a backfill over a period that included them and query every serving dataset for their key. If anything returns a row, the suppression list is not being consulted where it must be. Run it on a schedule, not once.
- It misses copies the query does not cover — logs, tickets, extracts, notebooks and anything outside the platform — and it misses inference: a subject removed from a dimension may still be identifiable from an unusual combination of facts.
- A completeness check per request is the other half: assert that every dataset named by lineage as containing the subject key returns zero rows for it, and that snapshot expiry has run on each since the rewrite (Reconciliation).
- Erasure has a deadline, and the deadline is the freshness requirement: the platform must be able to complete the full two-phase removal within it, including snapshot expiry and compaction, which are scheduled jobs rather than immediate operations.
- Batching trades fulfilment latency for cost. A daily batch is usually well within any statutory window and turns a per-request rewrite into one rewrite; an hourly batch costs much more and buys nothing anyone asked for.
- The suppression list must be effective immediately, before any rewrite completes, because ingestion continues while the deletion is being processed. Suppression is the low-latency half; rewrite is the high-latency half.
- A new dataset containing subject data must be added to the deletion scope, and it will not be unless scope is derived from lineage rather than maintained by hand (Data Lineage).
- Dropping a subject key from a dataset for minimization reasons makes it un-erasable at subject granularity, which is either fine — because it is now anonymous — or a serious problem, depending on whether the remaining columns identify. Decide which explicitly at the time (Data Minimization).
- A change in what counts as personal data extends the scope retroactively over data you already hold, so the scope query must be re-runnable rather than a one-time mapping (Data Classification).
- Erasure has no undo by design, so the risk to manage is over-deletion: a request matched on the wrong key removes the wrong subject, and the data is gone from the platform. Match on an authenticated identifier, and stage the deletion — mark, verify, then rewrite.
- If a deletion is incomplete rather than excessive, recovery is re-running it with a corrected scope, which is cheap. Incompleteness is the recoverable error and over-deletion is not, which should drive how the tooling is built.
- A restore from backup reinstates deleted subjects. Every restore must be followed by re-application of the suppression list, and this must be a step in the documented restore procedure rather than something someone remembers (Restore Testing).
What can go wrong
- The backfill resurrects the subject. The canonical failure of this lesson: deletion was a point-in-time cleanup and reprocessing rebuilt from a source that still contained them, or from raw that did.
- Deletion is logical only — delete markers written, snapshot expiry never scheduled — and the bytes remain readable to anything with file access indefinitely (Open Table Formats).
- The dimension row is deleted and the fact rows remain, leaving a full behavioural history keyed to a now-dangling surrogate (Fact Tables).
- A restore from backup silently reinstates every subject deleted since the backup was taken.
- The suppression list is consulted by the main ingestion path and not by a secondary one added later, so one source keeps re-introducing suppressed subjects.
- Per-subject deletion becomes a continuous compaction workload that competes with ingestion, and the team responds by batching so aggressively that the deadline is missed (File Compaction).
- The request is recorded as fulfilled based on the job exiting successfully rather than on verified absence — the domain's own thesis, applied to governance (The Pipeline Succeeded. The Data Is Wrong.).
- A model trained on the subject is left in production and the request is attested as complete, which is a misstatement rather than a technical failure.
- "We deleted the row, so the person is gone." You deleted one row in one copy. Raw, the log, the fact tables, the backups, the extracts and the trained model are all unaffected (PII in Pipelines).
- "A backfill is a technical operation." A backfill is a re-materialisation of data from a source, and without a suppression list it re-materialises people who asked to be erased. This is the specific reason deletion and reprocessing must be designed as one mechanism (What Backfills Break).
- "The table format supports deletes, so we are covered." It supports hiding rows immediately and removing bytes later. Only the second is erasure, and only the first happens automatically (Open Table Formats).
- "Backups are out of scope." They are a full copy of the pre-deletion state, and a restore reinstates it. Either the restore procedure re-applies suppression, or the deletion is undone by design.
- "Anonymising is the same as deleting." It preserves the aggregate history and removes the link, which is a different guarantee with a different legal status. Do not attest one having done the other.
- "The deletion job succeeded, so the request is fulfilled." The job exiting zero says the code ran. Verified absence in every dataset in scope, plus completed snapshot expiry, is the evidence (The Pipeline Succeeded. The Data Is Wrong.).
- Deletion is the mechanism that tests whether every other mechanism in this module was real. It fails precisely where classification was incomplete, where the copy inventory was unknown, and where retention was configured on a path nothing writes to any more.
- The suppression list is the durable artefact — more important than any individual deletion, because it is what makes the erasure survive the platform's own habit of rebuilding everything from source (Keeping Raw History: The Recovery Position and the Liability).
- Proof of erasure is a dataset with a schema, a grain and a retention, and it is itself sensitive: a record that a specific person requested erasure is personal data about them. Keep it minimal and keep it as long as the attestation must stand (Audit Logs for Privileged Actions).
- Where erasure is genuinely impossible — a trained model, a partner's copy, an immutable archive — record it as residual risk with an owner and a mitigation. That is defensible; an attestation that quietly excludes it is not (Risk, Residual Risk and Honest Reporting).
Operating it
- Per-request completion state, broken into logical delete, physical rewrite and snapshot expiry per dataset. A request is not done until the third column is green everywhere (Pipeline Observability).
- Time from request to verified absence, tracked against the deadline. The distribution matters more than the mean, because the tail is where the compliance failure is.
- Suppression-list hit rate at ingest. A non-zero, non-decreasing rate is expected and healthy; a sudden drop to zero usually means a path stopped consulting it rather than that sources stopped sending.
- Count of datasets in the lineage-derived scope versus datasets actually processed per request, which is the coverage measure and the one that catches a new dataset nobody added.
- Rewrite volume per deletion batch — files rewritten and bytes moved — because that is the number that decides whether the current batching interval is sustainable (What Actually Drives Data Platform Cost).
- At 10x requests, per-request processing must become batched, and the batch interval becomes a compliance parameter rather than an engineering convenience.
- At 100x, deletion becomes a continuous rewrite workload sharing capacity with compaction and ingestion, and it must be scheduled as a first-class pipeline with its own SLO (Pipeline SLOs).
- At high dataset counts, lineage-derived scope is the only thing that keeps up; a maintained list of affected datasets fails first and fails silently.
- Layout interacts strongly with scale here: if the subject key is also a clustering key, affected rows are concentrated and rewrites are cheap; if it is scattered uniformly, every deletion touches every file (Clustering and Sort Order).
- Deletion cost is driven by files touched, not rows removed. One subject scattered across a thousand files costs a thousand rewrites; a thousand subjects concentrated in ten files cost ten. Physical layout therefore decides deletion cost, which is an argument nobody makes when choosing a partition key (Partitioning).
- Batching is the main lever and it converts many rewrites of the same files into one, at the cost of fulfilment latency (File Compaction).
- Crypto-shredding moves the cost from rewrite time to key management: many keys to store, rotate and audit, and a decrypt operation on every read of the protected column (Key Management and Encryption at Rest).
- Verification is not free either. Proving absence means querying every dataset in scope per request, and at high request volume that becomes a scheduled reconciliation job rather than a per-request query (Scan Cost).
- Keeping the subject key everywhere makes erasure possible and increases identifiability, which is exactly what minimization argues against. Both are correct and the resolution is per dataset: keep the key where erasure is required, drop it where the remainder is genuinely anonymous (Data Minimization).
- Crypto-shredding makes erasure tractable across immutable and unreachable copies, and makes key management a correctness dependency: a lost key is an unrecoverable dataset, and a key hierarchy per subject is a real system to operate.
- Batching is cheaper and delays fulfilment. The interval must be chosen against the deadline with margin for the physical phase, which is the part people forget to include.
- Anonymisation preserves analytical history and is a weaker guarantee than deletion; whether it satisfies the obligation is a legal question and not one the platform can decide.
- Retaining the proof — which datasets were scanned, what was found, when it was removed — is itself a dataset about erased subjects, and it must be minimal enough not to become the record you were asked to erase.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe two-phase structure — rewrite every enumerable copy, then suppress so reprocessing cannot restore it — follows from the fact that analytical platforms rebuild derived data from retained sources, and applies to any platform that can replay or backfill.
- FORMAT-SPECIFICIceberg, Delta and Hudi all separate logical deletion from physical removal, with position and equality deletes hiding rows until compaction and snapshot expiry rewrite the files; a plain directory of Parquet files has no delete mechanism at all and requires an explicit rewrite of every affected file.
- BROKER-SPECIFICRemoving a subject from a log is not a delete: on a compacted topic it is a tombstone that retains the key until compaction runs, and on a time-retained topic it is simply waiting for the horizon, so the broker's retention model determines whether erasure there is possible at all.
- ORG-SPECIFICWhat erasure legally requires, whether anonymisation satisfies it, which datasets are in scope and what deadline applies are jurisdictional determinations; the engineering is the same everywhere and the acceptance criteria are not transferable between regimes.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the restore runbook, which is where the post-restore re-application of the suppression list must live. A restore that reinstates erased subjects is a correct restore and an incorrect outcome, and only the runbook can reconcile the two.
- — Distributed Systems owns why a delete is not simultaneous across replicas and regions, and why a copy in another region can serve a deleted subject after the primary has removed them.