ContractsGENERALTOOL-SPECIFICFORMAT-SPECIFIC

Backward Compatibility

The producer ships first: data written under the new schema must still be readable, and still correct, for consumers that have not upgraded. State it as writer-and-reader, because the word itself is used in opposite directions by different communities.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

If the producer deploys its schema change today and no consumer changes anything, does every consumer still get a correct answer tomorrow?

Who needs this

The consumers who have not upgraded, which is all of them at the moment the producer deploys. In any organisation larger than one team, the producer's release and the consumers' releases are not coordinated, so there is always a window in which new data is being read by old code. Backward compatibility is the property that makes that window uneventful.

What one row is

The unit is one field across one version boundary, evaluated from one reader's position. The same change is backward compatible for a reader that names its columns and breaking for a reader that asserts an exact column set, so the property is not a property of the change alone.

The obvious build

Reason about compatibility using the words. "This change is backward compatible" gets said in a review, everyone nods, and the change ships. Nobody notices that two people in the room meant opposite things by it, because both of their meanings are in common use and neither is wrong (Versioning: What a Version Even Promises).

Why it breaks

One engineer means "old consumers keep working" and another means "the new code can read the old data". They agree the change is backward compatible, ship it, and one of the two things is false (Schema Registry).

How it breaks with real data

One engineer means "old consumers keep working" and another means "the new code can read the old data". They agree the change is backward compatible, ship it, and one of the two things is false (Schema Registry).
The team configures a schema registry with the compatibility mode whose name matches the phrase they have been using, and gets the opposite rule set enforced — permitting exactly the changes they intended to forbid.
A field is removed. The producer verifies that the *new* pipeline reads fine, which is a statement about the other direction entirely, and every consumer still on the old schema breaks the following morning (Removing Fields Without Removing Consumers).
A type is widened from a 32-bit to a 64-bit integer. New data is written with values that fit the wider type, an old consumer reads them into the narrower one, and the overflow is either an error or, in a permissive engine, a wrapped value that looks like a real number.
An enum gains a value. Nothing about the schema changed, so every compatibility check passes, and every old consumer that branched over the previous value set silently reclassifies the new one (Enum Evolution: The New Value That Broke Old Clients).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Compatibility is always a relation between a writer schema (the shape the data was produced with) and a reader schema (the shape the consumer expects). Backward compatibility, as this lesson uses the term, is the case where the writer is new and the reader is old.
The mechanism that makes it work is reader tolerance. A reader that projects the fields it named and ignores everything else survives additions. A reader that asserts an exact column set, or that hashes the whole record, or that maps by position, does not — even for an addition (Unsafe Deserialization).
A self-describing format makes the relation explicit: the file or record carries the writer schema, the consumer supplies the reader schema, and the runtime resolves the two field by field. This is why the property is discussed most precisely in the Avro world — the resolution is a named algorithm rather than an accident (Avro).
A schema registry turns the relation into a gate: before a producer may register a new version of a subject, the registry checks the candidate against the previous version under a configured compatibility mode and refuses the registration if it violates it (Schema Registry).
The direction determines who must deploy first. If a change is compatible in this direction, the producer can deploy whenever it likes and consumers migrate at their own pace. If it is not, consumers must be migrated first — which is only possible if the change is compatible in the *other* direction, and if it is compatible in neither, you need two versions running side by side (Forward Compatibility).

Say it as writer and reader, or do not say it at all

TOOL-SPECIFICThe mode names above are a registry vocabulary, not a property of data. If your platform enforces compatibility somewhere other than a registry — in producer CI, at an ingestion gate, in a dbt contract — those names may not appear at all, and the writer/reader sentence is the only thing that transfers.

This is the one lesson in the module where the terminology is the subject rather than the packaging. Two large communities use these words in opposite directions, both consistently, both for defensible reasons, and a team that mixes them will configure the wrong enforcement and believe it is protected.

In the API and general-software tradition, compatibility is described from the point of view of the thing being changed: a backward-compatible change is one that does not break existing clients. The new server accepts old requests; the new producer emits data old consumers can still read. That is the sense this lesson uses, because it is the sense in which a data team asks "can I ship this".

In the Avro and schema-registry tradition, compatibility is described from the point of view of the schema doing the reading. A schema is backward compatible when it can read data written under the previous schema — new reader, old data. That is a different direction, and it is the direction this domain's Forward Compatibility lesson covers.

Neither community is going to change, so the only defence is to stop using the adjective on its own. Say which schema wrote the data and which schema is reading it. Every sentence below does exactly that, and the table maps the vocabulary so you can translate a policy rather than argue about it.

This lesson's direction: the producer ships first

UserLLMAgentToolDataDecisionHumanGuardrail

Direction, stated unambiguously	Who deploys first	API / general-software name	Schema-registry mode name	Changes that preserve it
New schema writes, old schema reads	The producer	Backward compatible change	FORWARD	Add a field old readers ignore; remove a field no reader requires and for which readers have a default.
Old schema writes, new schema reads	The consumer	Forward compatible change	BACKWARD	Add a field with a default so old records can be filled; remove a field the new schema no longer needs.
Both directions hold	Either, in any order	Fully compatible change	FULL	Add or remove an optional field that has a default on both sides. Little else.
Neither direction holds	Nobody, safely	Breaking change	NONE, or a rejected registration	Rename, retype, narrow an allowed-value set, or change a field's meaning. Requires a version and a migration window.

Product detail — verify current documentation

Confluent Schema Registry has used compatibility modes named BACKWARD, FORWARD, FULL and their TRANSITIVE variants for many years, and its BACKWARD mode means a consumer on the new schema can read data written under the previous one — the opposite of what the API world calls a backward-compatible change. Verify the current documentation before configuring a subject, and configure from the writer-and-reader sentence rather than from the name.

Which changes survive a new writer and an old reader

With the direction pinned down, the rules are short. An old reader looks for the fields it knows about. It survives anything that leaves those fields present, correctly typed and correctly populated, and it breaks on anything else.

The Avro pair below makes the resolution concrete. The reader schema is v1; the writer schema is v2. Resolution runs field by field: shipping_method is present in the writer and absent from the reader, so the reader skips it. Everything the reader declared is still present in the writer with a matching type, so every read succeeds. The producer can deploy this without telling anyone — which is exactly the property being described.

Now change one thing: delete currency from the writer. Resolution now needs a value for a field the reader declared and the writer did not send. Avro can supply the reader's default if it has one, and errors if it does not — so whether a removal is safe in this direction depends on a default in the reader, which is a property of code the producer does not control. That is the honest reason removals require notice rather than confidence.

Add a field — preserved, for readers that skip unknown fields. Broken for strict targets, positional readers and whole-record hashes.
Remove a field — preserved only if every reader either does not declare it or has a default for it. That is a property of consumer code, so it needs notice (Removing Fields Without Removing Consumers).
Rename a field — never preserved. It is a removal and an addition, and only a human knows they are related (Expand and Contract Migrations).
Change a type — not preserved in general. Some widenings resolve; most narrowings do not, and a permissive engine will produce null rather than raise (Breaking Schema Changes).
Add an enum value to a string field — invisible to every structural check, and breaks every old reader that branches on the previous set (Enum Evolution: The New Value That Broke Old Clients).
Make a non-null field nullable — structurally an addition of null to the domain, and a real break for every reader that assumed non-null (Nullability & Defaults).
Change what a field means — no schema difference at all, so no check in this lesson can see it (Semantic Changes).

Reader schema (v1, unchanged consumer) and writer schema (v2, deployed producer)

1// READER — what the consumer still expects
2{
3  "type": "record", "name": "OrderPlaced",
4  "fields": [
5    {"name": "order_id",     "type": "string"},
6    {"name": "placed_at",    "type": "long", "logicalType": "timestamp-millis"},
7    {"name": "amount_minor", "type": "long"},
8    {"name": "currency",     "type": "string", "default": "EUR"}
9  ]
10}
11 
12// WRITER — what the producer now emits
13{
14  "type": "record", "name": "OrderPlaced",
15  "fields": [
16    {"name": "order_id",        "type": "string"},
17    {"name": "placed_at",       "type": "long", "logicalType": "timestamp-millis"},
18    {"name": "amount_minor",    "type": "long"},
19    {"name": "currency",        "type": "string", "default": "EUR"},
20    {"name": "shipping_method", "type": ["null", "string"], "default": null}
21  ]
22}
23 
24// Resolution, field by field, from the READER's position:
25//   order_id        present in both, same type        -> read
26//   placed_at       present in both, same type        -> read
27//   amount_minor    present in both, same type        -> read
28//   currency        present in both, same type        -> read
29//   shipping_method present in writer only            -> SKIPPED, silently
30//
31// The old consumer parses every record and never learns that orders now
32// carry a shipping method. That is compatibility, and it is also blindness:
33// if "revenue" is about to be redefined per shipping method, this consumer
34// will keep computing the old definition perfectly.

The last comment is the point. Structural compatibility says the read succeeds. It says nothing about whether the consumer's answer is still the answer anyone wants.

When the change is not compatible in this direction

Sooner or later a change is required that no amount of tolerance will absorb. The useful move at that point is to stop asking "is this compatible" and start asking "in which order can these deploys happen, and what runs in the meantime".

There are only four answers, and the criteria that select between them are the number of consumers, how quickly the slowest one moves, and whether the producer can afford to emit two shapes at once. There is no winner here — a two-version window is right for a widely consumed dataset and absurd for one with a single internal reader.

Notice that three of the four options require knowing who the consumers are. That is the recurring precondition of this entire module, and it is why lineage is infrastructure rather than documentation (Impact Analysis).

A field must be renamed, retyped or removed

Who can deploy first, and what runs during the window?

Expand and contract

when The change can be expressed as an addition now and a removal later. Nearly all renames and most type changes can.

cost Both fields exist for the length of the window, the producer writes both, and someone has to actually come back and do the contract step. Deprecated fields that were never removed are the standard residue of this approach (Expand and Contract Migrations).

Migrate consumers first

when The change is compatible in the other direction — a new reader can handle old data — and the consumer set is small and known.

cost The producer waits for the slowest consumer. With more than a handful of consumers this is a schedule nobody controls (Forward Compatibility).

Publish a new dataset version

when The change is compatible in neither direction, or the consumer set is too large to coordinate at all.

cost Two datasets, two pipelines and roughly double the storage and compute for that dataset until the old one is retired — plus the discipline to actually retire it (API Migration: Running the Change End to End).

Coordinated cut-over

when One consumer, one producer, same team, and a maintenance window is acceptable.

cost A pause in the data and a rollback plan that has to cover data written under the new shape. Cheap at small scale and unavailable at any other (Rolling Back Data).

Two ways to remove a field

Remove it and announce it

Post in a channel that `amount_cents` will be dropped, wait two weeks, drop it. Consumers who read the message migrate; consumers who did not read it, or who are a scheduled query nobody owns, break on the morning of the drop.

Stop populating, watch, then remove

Announce, then keep the column and set it to null for new records while continuing to serve history. Watch reads of that column in query logs. Remove it only once reads have gone to zero and stayed there through a full monthly cycle.

An announcement measures who was listening; a read counter measures who is actually depending on the field, including the consumers nobody knew existed. The nulling step also converts the failure from "column not found" into a value-level anomaly that a null-rate monitor will surface — which is a worse failure in isolation but a much better one when it happens during a window you are watching.

How to build it

Most important first.

Never say "backward compatible" in a design document without the sentence that disambiguates it: data written by the new schema, read by a consumer on the old one. That sentence costs eight words and removes an entire class of expensive misunderstanding.
Prefer additive change. Adding a field that old readers can ignore is the one change that preserves this direction for nearly every consumer, which is why almost every safe migration is expressed as an addition followed later by a removal (Expand and Contract Migrations).
Before removing or retyping a field, establish who reads it — from query logs and lineage, not from asking around — and treat every one of them as a consumer who has not upgraded (Impact Analysis).
Make readers tolerant on purpose. Project named columns, never SELECT * into a fixed-schema target, and never hash a whole record when you mean to hash a key (Three Models, Not One).
Encode closed sets as closed sets in the contract, and test them at the boundary, because an enum addition is invisible to every structural compatibility check (Enum Evolution: The New Value That Broke Old Clients).
Where the registry is the enforcement point, configure the compatibility mode from the reader/writer sentence, not from the name of the mode, and write a comment in the configuration saying which direction was intended (Schema Registry).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

When this direction holds, it guarantees that a consumer on the old schema can still parse the new data. It does not guarantee the consumer's answer is still right.
It guarantees nothing about fields the reader never declared. A tolerant reader ignoring a new field is compatible and also blind — if the new field carries a distinction the metric now depends on, the old consumer computes a stale definition perfectly (Semantic Changes).
Under the transitive variants of a registry's compatibility modes, the guarantee extends across all previous versions rather than only the immediately preceding one. Under the non-transitive variants it holds only against the last version, which means a chain of individually compatible changes can be collectively incompatible.
Nothing here guarantees anything about historical data. That is the other direction, and it is a separate check (Forward Compatibility).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check is a compatibility test in the producer's pipeline: resolve the candidate schema against the schema each known consumer is running, and fail the build if any of them can no longer read the output (Contract Tests Between Services).
It misses tolerant readers that parse successfully and compute the wrong thing — the addition of a field that changes what the old definition means is compatible by every structural test and wrong by every business one.
It also misses consumers you do not know about, which is the usual reason a compatible change breaks something. The test is only as complete as the consumer list, and the consumer list should come from lineage rather than from memory (Data Lineage).

Freshness

This direction is the one that buys deployment independence, and deployment independence is what keeps freshness stable during a migration: the producer never has to wait for the slowest consumer before shipping.
When it does not hold, the change becomes a coordinated release, and the data is either paused or wrong for the duration of the coordination.
A registry gate that rejects a producer's schema registration stops production of that data entirely. That is the correct outcome and it is also a freshness incident, so the rejection needs to reach the producing team quickly rather than sitting in a log (Contract Enforcement).

When the schema or meaning changes

Compatibility policy itself evolves. A dataset usually starts with no policy, acquires one when it first breaks someone, and tightens it as consumers multiply. Tightening is a breaking change for producers, so it deserves the same notice period as a schema change.
A long-lived dataset accumulates versions, and a policy that only compares against the immediately previous version permits gradual drift that no single check ever refuses. Transitive checking is the answer and it costs more.
Eventually some change is compatible in no direction. That is not a failure of the policy — it is the case the policy exists to detect, and the answer is a new dataset version and a migration window (API Migration: Running the Change End to End).

How to re-run this safely

If an incompatible change reached consumers, the fastest repair is usually at the consumer boundary rather than the producer: a staging model that coalesces old and new field names, or casts both to a common type, restores every downstream model at once (Model Layering).
The producer-side repair is to re-emit the affected range under a schema old consumers can read, which requires the producer to still have the data and to be able to replay it (Replay from the Log).
Data written during the incompatible window is still correct on disk if raw was landed unmodified. It is the *interpretation* that failed, and interpretations are re-runnable (Keeping Raw History: The Recovery Position and the Liability).

What can go wrong

Failure modes

The phrase is used without the direction and two teams act on opposite understandings.
A registry configured with the mode whose name matches the phrase rather than the intended rule, enforcing the opposite policy while looking correct in review (Schema Registry).
A compatibility check that runs against the previous version only, allowing a sequence of small steps to strand a consumer three versions back.
A consumer that parses fine and is now semantically stale, which every structural check reports as healthy (Semantic Changes).
The consumer list used by the check being out of date, so the test passes for the three consumers you know and fails for the fifth.
A tolerant reader used as an excuse not to have a policy — tolerance protects against additions and nothing else.

Misreads

"Backward compatible means the new code can read the old data." That is the other direction, and it is what a Confluent-style registry means by its BACKWARD mode. Both usages are in wide circulation; only the writer-and-reader sentence is unambiguous (Forward Compatibility).
"The compatibility check passed, so the change is safe." The check verifies parsing. It cannot verify that the consumer's query still means what it meant, and a compatible change can absolutely alter a metric (Semantic Changes).
"Adding a field is always compatible in this direction." It is compatible for readers that ignore unknown fields. It breaks strict targets, exact-column-set assertions, whole-record hashes and positional readers (Schema Evolution).
"We use Avro, so we get this for free." Avro gives you a resolution algorithm and a place to put defaults. It does not choose your compatibility policy, does not know your consumers, and has nothing to say about enum values in a string field (Parquet vs Avro).

Operating it

How you see it in production

Registered schema versions per subject, with the timestamp and the author. This is the changelog consumers actually need and it is generated rather than written (Metadata: Technical, Operational and Business).
The set of schema versions currently in use by live consumers, which is the input the compatibility test needs and the number nobody tracks until the first incident.
Rejected schema registrations per producer, which is the leading indicator that a team is trying to make a change the policy forbids and will eventually route around (Contract Enforcement).
Per-column null rate immediately after any producer deployment, as the cheap general-purpose detector for a change that was declared safe and was not (The Data Quality Dashboard).

What changes at 10x and 100x

At one consumer, this property is unnecessary — coordinate the deploy and move on. It becomes load-bearing at the point where the producer cannot enumerate its consumers from memory.
At 10x consumers, the probability that all of them have upgraded before the producer ships approaches zero, so the producer must assume an old reader always exists (Data Contracts).
At 100x datasets, checking compatibility by hand is impossible and the property must be enforced by a gate in the producer's pipeline or it does not exist (Schema Registry).

What drives cost here

The direct cost is engineering discipline rather than compute: additive-only change means carrying deprecated fields for as long as any consumer still reads them, which is storage and cognitive load.
Running the compatibility check itself is negligible — it compares two schemas, not two datasets.
The cost that surprises teams is the deprecation tail. Fields kept "until the last consumer migrates" outlive the engineers who added them, and a dataset with forty columns of which twelve are deprecated is a normal outcome of doing this correctly for several years (Dataset Documentation).

What this approach costs

Preserving this direction constrains the producer. They cannot remove, rename or retype a field on their own schedule, which is precisely the freedom the property is trading away in exchange for consumers who do not get paged.
Tolerant readers are what make the property achievable, and tolerance is also how a consumer misses a field that mattered. You cannot have a reader that ignores unexpected fields and also alerts on them without deciding, per field, which behaviour you want.
Transitive compatibility is stronger and more restrictive: it forbids changes that are individually reasonable because some consumer might still be four versions back. Whether that is worth it depends entirely on how slowly your slowest consumer moves.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe relation between a writer schema and a reader schema is universal and is the only formulation that survives translation between communities. What is not universal is the vocabulary: the API world and the schema-registry world attach the words "backward" and "forward" to opposite directions.
TOOL-SPECIFICSchema registries enforce this as a named mode on a subject, and the mode names do not agree with API-world usage. A registry mode also compares only the schemas registered in that registry, so a consumer reading the data through a warehouse table rather than through the registry is outside the check entirely.
FORMAT-SPECIFICAvro resolves reader against writer per record and can supply a default for a field the writer omitted, so the direction is well defined; Protobuf achieves much of the same through field numbers and unknown-field retention; JSON on its own has no resolution step at all, so every guarantee has to come from a validator you supply.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Backendexpand-contract-migration contract-tests deserialization serialization-basics

API Designbackward-compatibility consumer-driven-evolution removing-fields enum-evolution deprecation versioning api-migration

Domains that do not exist yet

— DevOps / Production Engineering owns the release ordering this lesson depends on: which artefact ships first, how long both are live, and what a rollback does to data already written under the new shape.