Schema Evolution
Schemas change constantly. Adding, removing, renaming and retyping a field are four different risks with four different blast radii — and only one of them is routinely safe.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A field is about to change upstream. Which consumers break, which keep working, and which keep working while producing the wrong answer?
Every reader of the dataset, but unevenly. A consumer that selects three named columns is affected by a change to one of those three. A consumer that does SELECT * into a strict target is affected by every change including additions. A consumer that branches on a value set is affected by changes nothing in the schema records. Knowing which kind of consumer you have is most of knowing what a change will do.
The unit of evolution is one field, across two schema versions. Reasoning about "the schema changed" produces no useful answer; reasoning about "status gained a sixth allowed value while amount_cents was renamed" produces two different answers, one of which is invisible to every schema check ever written.
Let the schema follow the source. Whatever shape the producer emits this week is the shape that lands, transformations are updated when something breaks, and the team treats each break as a small maintenance task. This is genuinely how most platforms start and it works while there is one producer and one consumer who sit near each other.
A column is renamed. The ingestion job, which infers schema from the file, happily lands a new column and stops populating the old one. Nothing errors. The old column is still present in history, is now always null, and every model that reads it reports zero for recent periods (Breaking Schema Changes).
- A column is renamed. The ingestion job, which infers schema from the file, happily lands a new column and stops populating the old one. Nothing errors. The old column is still present in history, is now always null, and every model that reads it reports zero for recent periods (Breaking Schema Changes).
- A field changes from integer to string. The transformation casts it back, the cast produces null on failure rather than raising, and the row survives with nothing in it. Row counts still reconcile (Nullability & Defaults).
- A field is added. This is the safe change — except for the consumer whose target table is strict, or whose
SELECT *now writes an extra column into a downstream table with a fixed schema, or whose deduplication hashes the whole record and now sees every row as new (Deduplication). - A field is removed after a team checked that "nobody uses it" by asking around. Query logs would have shown four dashboards and a monthly export; asking around showed none (Impact Analysis).
- An enum gains a value. No schema anywhere changed if the field is typed as a string, and every
CASEexpression downstream silently reclassifies the new value (Enum Evolution: The New Value That Broke Old Clients). - History becomes internally inconsistent: partitions written before the change have one shape, partitions after have another, and a query spanning both either fails or — worse — succeeds with nulls for the periods where the field did not exist (Full Refresh vs Incremental).
What is actually happening
- Schema evolution is a compatibility question between a writer and a reader, and it only has an answer once you say which is which. The same change is safe in one direction and fatal in the other, which is why the words "backward" and "forward" cause so much trouble here (Backward Compatibility).
- Different layers of the stack handle a change at different times. A self-describing file format carries its own schema, so a reader can resolve differences per file. A warehouse table has one schema for all its data, so a change has to be applied to the table itself. A stream has no schema at all unless a registry supplies one (Avro).
- The dangerous property of most data tooling is that it is permissive by default. Inference-based ingestion adds columns it has never seen, casts that fail produce null instead of raising, and missing columns read as null. Every one of those defaults converts an error into a silently wrong value (Breaking Schema Changes).
- A rename is not one change. To every system that identifies fields by name — which is nearly all of them — it is a removal plus an addition that happen to be related, and only a human knows they are related (Column-Level Lineage).
- The blast radius is set by lineage, not by the change. A field near the source with fifty downstream models has a different risk profile from the same change to a leaf table, and the only way to know which you have is a dependency graph (Data Lineage).
Five changes, in ascending order of danger
Schema changes are discussed as if they were one category, which is why they are handled badly. They are five categories with genuinely different risk, and the ordering below is stable across stacks because it follows from how systems identify fields — by name, positionally, or not at all.
The critical property is not whether a change breaks something. It is whether the break is loud. A change that causes a job to fail is cheap: the orchestrator tells you, the data does not move, and you fix it. A change that causes a job to succeed with different values is expensive, because the only detector left is a human who notices the number looks odd.
The schemaDiff below works through a rename, which is the change most people underestimate. Read the silent column: two of the five consumers get an error and three get an answer. The three who get an answer are the incident.
- order_id: string
- placed_at: timestamp
- amount_cents: integer
- currency: string
- status: string
- order_id: string
- placed_at: timestamp
- amount_minor: integer
- currency: string
- status: string
change The producer renames amount_cents to amount_minor in a single migration. The schema file in their repository is updated in the same commit. No consumer is notified.
| Consumer | Effect | How it shows up |
|---|---|---|
| Strict ingestion job with a declared schema | Rejects the batch: a required column is missing and an unknown one is present. The pipeline stops and someone is paged within the hour. | Loudly — it raises |
| Transformation model selecting `amount_cents` by name | Fails to compile or errors at run time — the column does not exist in the new arrivals. | Loudly — it raises |
| Inference-based ingestion into a permissive table | Adds amount_minor as a new column and keeps amount_cents, now always null. Both columns exist; neither is complete; every historical query mixes them. | Silently — no error, wrong result |
| Revenue model summing `amount_cents` over the permissive table | Returns zero for every period after the change and the correct figure before it, so the dashboard shows a business that stopped trading on a Tuesday. | Silently — no error, wrong result |
| BI extract doing `SELECT *` into a spreadsheet | Gains a column, keeps a stale one, and every downstream formula that referenced a column position is now off by one. | Silently — no error, wrong result |
add a nullable field loud only for strict targets and *-selects; otherwise safe remove a field loud for named readers; invisible until they run add an enum value NEVER loud — the schema did not change at all rename a field silent for permissive ingestion, loud for strict change a field's type usually silent: the cast returns null, not an error change a field's meaning silent by construction; no tool can ever see it
Which hop absorbs the change, and which passes it on
A schema change enters at the producer and travels the same path the data does. Each hop makes one decision: absorb the change, reject it, or pass it through. The design question is not "how do we prevent changes" — you cannot — but "at which hop do we want to find out".
The right answer for almost every platform is: land permissively, promote strictly. The raw layer accepts whatever arrived, because its job is to be evidence and evidence that has been edited is not evidence. The boundary from raw into modelled data is where conformance is asserted, because that is the last point at which failing is cheap (Raw, Staging, Curated: Layers by Purpose).
Read the guarantees column below and note how quickly the promises run out. By the time data reaches the serving table, nothing in the chain has promised that a field still means what a consumer thinks it means — and the dashboard, which promises nothing at all, is where the change becomes a business decision.
- 1Producer service
Emits the record with the new field name, having updated its own schema file honestly.
guarantees That the payload matches the producer's current schema. Nothing about anyone else's expectations.
fails by Shipping a rename as a single migration, because in the producer's own codebase that is exactly what it is.
- 2Producer CI contract test
Compares the emitted payload against the agreed contract and fails the build on a breaking change.
guarantees That a declared breaking change cannot reach production unnoticed — if the contract declares this field.
fails by Not existing, or covering only the fields someone thought to declare (Contract Tests Between Services).
- 3Ingestion boundary
Validates the arriving batch against the expected schema and routes non-conforming batches to quarantine.
guarantees That structurally invalid data does not enter the platform.
fails by Being configured to infer rather than to assert, which turns a rejection into a new column.
- 4Raw landing
Writes the payload exactly as received, including fields nobody expected.
guarantees That whatever arrived is recoverable, which is what makes every later mistake fixable.
fails by Being "cleaned" on the way in, destroying the only proof of what the producer actually sent.
- 5Staging model
Casts, renames and normalises raw into a stable internal shape.
guarantees Only what its tests assert. A cast here is the single most common place a type change becomes null.
fails by A permissive cast succeeding on unparseable input (Breaking Schema Changes).
- 6Curated model
Joins and aggregates into the facts and dimensions consumers query.
guarantees The grain and the column set it declares — if it declares them.
fails by Aggregating a column that is now entirely null and reporting the sum as zero rather than as unknown.
- 7Serving table and BI
Answers consumer queries.
guarantees Nothing about upstream meaning. It renders what it is given.
fails by Presenting a confident number that changed because a column was renamed three systems upstream (Stale Dashboards).
The cheapest place to fail is the earliest one. Every hop to the right of the failure point costs more to unwind, and the rightmost hop costs a conversation with whoever acted on the number.
What an unannounced change looks like from the on-call side
You will usually meet a schema change as a symptom rather than as a notification. The table below is organised the way an incident actually arrives: the trigger is something you observe, the cause is what you have to work back to, and the response is what stops the bleeding rather than what fixes the root cause.
One pattern is worth internalising: a column that goes from zero nulls to all nulls is a schema change until proven otherwise. It is the shared signature of a rename, a failed cast, a dropped column and a producer that stopped populating a field, and it is visible from your own side without any cooperation from the producer.
The other pattern is the absence of a symptom. If a change is genuinely silent — an enum value, a meaning change — nothing in this table will trigger. That is not a gap in the table; it is the boundary of what schema-level thinking can do, and it is why the next lessons are about compatibility direction and about meaning (Semantic Changes).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A model that has run for a year suddenly fails to compile. | The referenced column does not exist. | A field was removed or renamed upstream and the ingestion layer is strict enough to have reflected it. | The cheapest failure in this table. Coalesce the old and new names in staging, then chase the producer for the contract change and a deprecation window. |
| A column's null rate jumps to 100% on a specific day. | Aggregates over that column collapse toward zero; row counts are unchanged. | A rename absorbed by permissive ingestion, a failed cast, or a producer that stopped populating the field. | Compare the raw payload before and after the boundary date — raw is where the truth is. Do not fix forward until you know which of the three it was, because the backfill differs. |
| A period-over-period comparison shows a category that appeared from nothing. | A new value in a low-cardinality column, and every previous bucket slightly smaller. | An enum gained a value. No schema changed, so no schema check fired (Enum Evolution: The New Value That Broke Old Clients). | Add an allowed-value test so the next one alerts, then decide with the consumer which existing bucket the new value belongs to — that is a business decision, not an engineering one. |
| A query spanning several months errors on type mismatch. | Old partitions and new partitions disagree about a column's physical type. | A type change applied to the table going forward but never to history. | Read through a view that casts both sides to a common type explicitly, and schedule the history rewrite as its own project rather than doing it during an incident. |
| A downstream table gains a column nobody added. | A strict consumer of that table starts rejecting. | An additive change upstream travelling through a SELECT * transformation. | Name the columns in the transformation. A SELECT * in a model is a promise to propagate every future upstream change, made by someone who did not know they were making it (Data Engineering Anti-Patterns). |
| Nothing. No alert, no failure, no anomaly. | A consumer reports the number is wrong, four weeks later. | A change with an identical schema — a meaning change, a unit change, a scope change to what the producer includes. | There is no technical response. The defence had to exist before the change: documented semantics, an owner and a changelog consumers can see (Semantic Changes). |
How to build it
Most important first.
- Rank changes by their real risk before discussing them, and give each rank a different process: additive changes ship, removals require notice and a deprecation window, renames and type changes require the expand–contract sequence, and meaning changes require a new field name (Semantic Changes).
- Never rename or retype in place. Add the new field, dual-write both, migrate consumers, then remove the old one — the same expand-and-contract sequence used for database migrations, for the same reason. A type change is a rename with a worse failure mode, because the old readers do not disappear: they succeed and produce null (Expand and Contract Migrations).
- Make the ingestion boundary strict and the raw layer permissive: land whatever arrived, unchanged, but refuse to promote it into a modelled table if it does not conform. That keeps evidence of what really happened while still failing loudly (The Raw Landing Zone).
- Type enums as enums somewhere. If the transport cannot express a closed set, express it as an allowed-value test at the boundary, so a sixth value is an alert rather than a silent bucket reassignment (Data Tests).
- Version the dataset when a change cannot be made compatibly, and run both versions during the migration window. Two datasets for a quarter is cheaper than one broken quarter (Versioning: What a Version Even Promises).
- Record the schema version alongside the data, in partition metadata or in the record itself, so a query that spans a change can be reasoned about rather than guessed at (Metadata: Technical, Operational and Business).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A self-describing format guarantees that a reader can always determine the schema a given file was written with. It guarantees nothing about whether the reader can make sense of it (Parquet).
- Adding a nullable field is the only change that is close to universally safe, and even that is only safe for readers who name their columns.
- No format, registry or warehouse guarantees that a rename is recognised as a rename. That relationship exists only in a human's head and in whatever you wrote down.
- Nothing guarantees history is re-readable under the new schema. After a type change, old partitions still hold the old physical type, and whether a query across the boundary works depends entirely on the engine (Query Engines).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that catches most of this is a schema-drift check at the boundary: compare the arriving schema with the expected one and alert on any difference — added fields, removed fields, changed types, changed nullability (CDC and Schema Drift).
- It misses renames as renames, seeing only a removal and an addition; it misses enum growth entirely if the field is a string; and it misses every change where the schema is identical and the meaning is not (Semantic Changes).
- Pair it with a per-column null-rate monitor. A column that was 0% null and is now 100% null is the exact signature of a rename or a failed cast, and it is visible without knowing anything about the producer (The Data Quality Dashboard).
- Schema evolution done well costs latency at the change: a deprecation window is measured in weeks, and during it consumers see two fields where they expected one.
- Schema evolution done badly costs latency at the incident instead — the pipeline is paused while someone works out what changed, and the affected period is stale until a backfill completes (Backfills).
- A permissive ingestion layer preserves freshness at the cost of correctness: data keeps flowing at full speed and is wrong. A strict one trades freshness for correctness, deliberately.
- This lesson is the evolution one, so the recursive question is what happens when the *evolution policy* changes. Tightening a permissive pipeline is itself a breaking change for producers who have been relying on it accepting anything.
- A platform that has run permissively for years has history written under many undocumented shapes. Reconstructing which is a metadata archaeology exercise, and it is the argument for recording schema version with the data from day one.
- Compatibility policy should be per-dataset, not per-platform. A dataset with two internal consumers and one with eighty external ones cannot reasonably run the same change process.
- Recovering from an unnoticed schema change means reprocessing every affected partition from raw, which is possible exactly when raw was landed unmodified (Reprocessing vs Retrying).
- If the change was a rename, the fix is usually a coalesce across both column names in the staging layer for the transition period, applied to history as well as to new data (Model Layering).
- If the change was a type change that nulled values, the raw layer still holds the original strings and the values are recoverable. If the pipeline cast on ingest, they are not (Keeping Raw History: The Recovery Position and the Liability).
- Backfill in dependency order and validate before publishing, because a schema-driven backfill touches every model downstream of the changed field (Validating a Backfill Before You Publish).
What can go wrong
- Inference-based ingestion absorbing a rename as an unrelated new column, so both columns exist and neither is complete.
- A cast that produces null rather than raising, converting a type change into a silent zero (Breaking Schema Changes).
- A deprecation window that ends because a calendar reminder fired, not because anyone verified the old field was unused.
- A strict boundary that rejects a batch at 2 a.m. for an additive change that was actually safe, teaching the team to widen the check until it catches nothing (Quality Alerting).
- Dual-write during expand–contract diverging, so the old and new fields disagree and nobody notices which is authoritative.
- A query spanning the change boundary that succeeds and returns nulls for the older half, which reads as "the business was quiet then".
- "Additive changes are always safe." They are safe for consumers that name their columns. They break strict targets, whole-record hashes,
SELECT *into fixed-schema tables, and anything that asserts an exact column set (Removing Fields Without Removing Consumers). - "The ingestion job handled it, so it was fine." Permissive ingestion handling a change is not evidence the change was safe — it is the mechanism by which unsafe changes become invisible.
- "We can rename it, we will just update the models." You will update the models you know about. The dashboards, notebooks, exports and one scheduled email you do not know about are the ones that break (Data Discovery).
- "Schema evolution is a format feature." Formats resolve reader and writer schemas for you. They do not tell you whether the resolution produced something the consumer can use, and they have nothing at all to say about meaning (Parquet vs Avro).
Operating it
- A schema-version-per-partition column in your metadata store, so "when did this change" is a query rather than an investigation (Metadata: Technical, Operational and Business).
- Null rate per column per day. The single most informative chart for schema problems, and it needs no knowledge of the producer (The Data Quality Dashboard).
- Distinct-value count per low-cardinality column, which is how a new enum value announces itself (Distribution Tests).
- Column-level lineage from the changed field to every downstream model and dashboard — the impact query, run before the change rather than during the incident (Impact Analysis).
- At 10x consumers, informal notice stops working. The number of people who need to know exceeds the number anyone can remember, and the change process has to become mechanical (Data Contracts).
- At 10x datasets, hand-maintained schema documentation is already stale and the only viable source of truth is the one generated from the data itself (The Data Catalog).
- At 100x history, a type change stops being applicable to history at all within any reasonable window, and the answer becomes a new field plus a view that coalesces the two — permanently (Model Layering).
- A dual-write window costs storage for both fields and pipeline complexity for as long as it lasts. It is the cheapest correct option and it is still not free.
- A type change applied to history costs a full rewrite of every affected partition, which is proportional to retained bytes rather than to new data (What Actually Drives Data Platform Cost).
- Running two dataset versions in parallel roughly doubles the storage and compute for that dataset during the migration, and is almost always cheaper than the incident it replaces.
- Strictness costs availability of the pipeline and buys correctness of the data. There is no setting that gives both, and the honest design states which one this dataset prioritises.
- Expand–contract is slower than a rename and requires the producer to run two code paths. It is nonetheless the only sequence in which no consumer is ever broken by a change they did not know about.
- Recording schema version with the data costs a little metadata on every partition forever, and pays for itself the first time someone asks what a column meant in March.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe ranking of change classes by danger — add, remove, rename, retype, re-mean — holds everywhere, because it follows from how systems identify fields rather than from any product. What varies is which layer notices first and whether it raises or coerces.
- FORMAT-SPECIFICAvro resolves a reader schema against a writer schema per record and applies defaults for fields the writer omitted, so a missing field is a defined outcome; Parquet stores a schema per file and leaves reconciliation across files to the engine; JSON and CSV carry no schema at all, so every guarantee has to be supplied by something outside the file.
- ENGINE-SPECIFICWhether a failed cast raises or returns null is an engine decision and is the difference between a loud incident and a silent one. Several SQL engines offer both a strict and a permissive cast; which one your transformation tool emits by default is worth checking rather than assuming.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the deployment sequencing that makes expand–contract work in practice: shipping the producer change and the consumer change as separate releases, with a window between them and a rollback that does not strand data written under the new shape.