The question this answers
If everything except one store were deleted, which one would let me rebuild the rest?
A named source of truth guarantees a defined repair direction: for any disagreement between it and a derived store, the derived store is wrong and can be corrected without judgement. It guarantees nothing about how fresh the derived copies are — only which way the arrow points when they differ.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Any store can report a value with complete confidence; none can report whether it is the authoritative one. Authority is not a property a component can observe in itself — it is a fact about the system, assigned by design and enforced by write paths. A derived store that has been given a write endpoint looks exactly like a source of truth from inside, which is why the label must be maintained deliberately rather than inferred.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The reconstruction test
There is one question that identifies the source of truth without argument: if every store but one were deleted, which one would let you rebuild all the others? That store is the source of truth for the state in question. Everything you could rebuild from it was derived.
The test is useful because it is concrete and because it exposes the uncomfortable cases immediately. If a search index holds annotations that exist nowhere else, it is not fully derived — it is a source of truth for those annotations, and nobody has said so. If a cache holds a computed value whose inputs are no longer available, the cache is authoritative and irreplaceable, which is a serious and usually unintentional design.
Run it per field, not per store, because the answer differs within one store: a read model is derived for the entity fields it projects and authoritative for the view-specific metadata it accumulates. Those two kinds of data need different treatment — one is rebuildable, the other must be backed up — and mixing them in one store without labelling is how a routine rebuild becomes data loss.
store: order-search-index
order_id, status, total, customer_name -> rebuildable from orders-db
placed_at, line_items -> rebuildable from orders-db
---------------------------------------------------------------
ops_flag_reviewed (set by agents in the search UI) -> NOT rebuildable
ops_review_note (free text, 3 years of it) -> NOT rebuildable
verdict: this index is DERIVED for 6 fields and AUTHORITATIVE for 2.
* it is not safely droppable, despite being called a "search index"
* those 2 fields have no backup policy, because nobody classified
this store as holding authoritative data
* a rebuild "from the source of truth" would delete 3 years of
review notes, and the runbook says rebuilds are safeTruth, cache, and the trap of "the fresher one wins"
When two stores disagree, the instinct is to prefer the one that looks more recent, or more complete, or more plausible. All three instincts are wrong, and each produces a different bad outcome.
Preferring the fresher value means the winner is decided by timestamps taken on different machines, which [[clock-skew]] makes arbitrary — and consistently arbitrary, since a node with a fast clock wins every time. Preferring the more complete value rewards whichever store happens to accumulate extra fields, which is usually a derived store that has grown a second writer. Preferring the plausible value substitutes an engineer’s judgement under incident pressure for a system property, and it does not generalise to the next thousand records.
With a named source of truth, none of this arises. The owner is right by definition, even when it is the older value, even when it looks wrong. If the owner holds a value that is genuinely incorrect, that is a separate problem — a bad write, to be fixed by writing the correct value *to the owner* — and it does not change the direction of repair for the other stores.
| Rule applied | What actually decides | Failure it produces | |
|---|---|---|---|
| The fresher value winsassumption | Timestamp comparison | Clock skew between machines | One node with a fast clock wins permanently, silently |
| The more complete value winstypical | Field count | Which store grew extra writers | Rewards exactly the store that violated ownership |
| The plausible value winstypical | An engineer at 03:00 | Judgement under pressure | Unrepeatable, unauditable, does not scale past one record |
| The source of truth winsprotocol | A design decision made in advance | The declared owner | None for direction; a wrong owner value is a separate, fixable problem |
Where the source of truth quietly moves
Systems rarely start without a source of truth. They lose it, and there are recognisable ways.
The event log versus the database. With event sourcing the log is authoritative and the state store is a projection. Teams often start that way and then someone patches the state store directly, at which point the log can no longer rebuild it and neither store is authoritative. The tell is that a replay produces a different result from what is currently stored.
The cache that outlived its source. A value is computed from inputs that have since been deleted or changed. The cache is now the only place that value exists, and it is being treated as ephemeral. Anyone flushing that cache destroys data.
The external system. A payment provider, a SaaS CRM, an identity provider is authoritative for state you hold copies of. Your database is derived, and treating it as authoritative produces the classic incident: your records say the subscription is active, theirs say it lapsed, and theirs is the one that is true. Being derived for data about your own customers is normal and needs to be labelled, not resisted.
The spreadsheet. Business-critical values maintained by a person in a file, imported occasionally. It is genuinely the source of truth and it is on nobody’s architecture diagram, has no backup policy and no access control. This is more common than any engineer wants to believe.
- Replay of the event log produces a different result from the current state store — the log is no longer authoritative.
- A cached computed value whose inputs no longer exist — the cache is authoritative and treated as disposable.
- A partner or SaaS system holds the real answer while your database is treated as authoritative.
- A person maintains the values in a spreadsheet — real authority, no backups, no access control.
- A derived store has grown fields nobody else has, so a rebuild would lose data.
Making the answer usable rather than merely correct
Naming the source of truth pays off only if the naming is reachable when it matters. Three practices make it so.
Record it where the data is, not in a wiki page nobody opens during an incident: a comment in the schema, a field in the service catalogue, a header on the API that serves the derived copy. The failure mode of an undocumented answer is that it is reconstructed under pressure, badly.
Label derived responses as derived. An API returning a cached or projected value can say so — a freshness header, an as_of field, a stale: true marker. This turns an invisible property into something a consumer can act on, and it makes "is this current?" answerable without asking a human.
Test the reconstruction. The claim "this store is derived" is a claim that it can be rebuilt from the owner. Exercise that path on a schedule. A derived store whose rebuild has never been run is a source of truth that has not been recognised yet, and the recognition usually happens at the worst possible moment.
Key points
- The source of truth is the store from which everything else can be reconstructed.
- Run the reconstruction test per field, not per store — one store can be derived for some fields and authoritative for others.
- When stores disagree, the source of truth wins by definition, even when it looks older or less complete.
- Choosing the fresher value hands the decision to clock skew; choosing the more complete one rewards a second writer.
- Authority drifts: patched projections, caches that outlive their inputs, external systems, and spreadsheets.
- A derived store whose rebuild has never been tested is an unrecognised source of truth.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • For each piece of state, ask which store could rebuild all the others if the rest were deleted.
- • Name that store as the source of truth, per field where the answer differs within a store.
- • Route all writes to it and make every other copy read-only.
- • Label derived responses as derived, with freshness information the consumer can use.
- • Exercise the reconstruction path on a schedule so the derived label stays true.
- • On disagreement, repair from the source of truth without deliberation about which value looks better.
- • A derived store accumulates fields that exist nowhere else, quietly becoming authoritative.
- • The propagation path from the source of truth breaks and derived copies serve stale values indefinitely.
- • Someone writes directly to a derived store during an incident and the divergence becomes permanent.
- • An external system changes state without notifying you, and your derived copy is confidently wrong.
- • A reconstruction is attempted and fails, revealing that the source of truth cannot in fact produce the derived state.
- • Rebuild as data loss: the operator drops a "derived" search index to rebuild it and destroys three years of operator review notes that lived only there.
- • Skew decides the winner: the operator sees one service consistently overwriting another’s values and finds no rule anywhere — the merge prefers the newer timestamp and one node’s clock runs fast.
- • External truth ignored: the operator sees customers with active subscriptions being denied service, because the provider lapsed them and the local copy is treated as authoritative.
- • Replay divergence: the operator replays the event log into a fresh projection and gets different values from production, revealing that the state store has been patched directly.
- • Stale forever: the operator sees a derived store serving values four days old with no errors anywhere, because nothing compares it to the source and its consumer lag is unmonitored.
- • Spreadsheet outage: the operator discovers a business-critical mapping is maintained in a file on one person’s laptop, when that person is on leave.
- • A single source of truth removes the need to agree: there is nothing to negotiate when one store defines the value.
- • Propagation to derived stores is asynchronous and needs no agreement, which is what makes the pattern available during partitions.
- • Where an external system is authoritative, coordination becomes a contract question — how you learn of changes, and how quickly — and webhooks plus periodic reconciliation are the usual pair.
- • The expensive coordination is agreeing the map in the first place, especially where two teams both believe they own a field. That conversation is cheaper than the incident it prevents.
- • When the source of truth is unavailable, derived copies keep serving last-known values, which should be labelled as possibly stale rather than presented as current.
- • Writes are unavailable during that window — the deliberate cost of having one authority.
- • Divergence introduced during an outage persists until reconciliation runs; it does not resolve itself.
- • For externally owned state, your copy may be wrong at any moment with no local signal, so periodic comparison is the only defence.
- • Detect: compare each derived store against the source of truth on a schedule — nothing else surfaces this class of problem.
- • Contain: block writes to derived stores as soon as a second writer is suspected, or repairs will be undone.
- • Recover: repair derived copies from the source, or rebuild where that is cheaper and the rebuild path is tested.
- • Reconcile: replay any corrections that were applied to derived stores through the source of truth so they survive.
- • Verify: re-run the comparison to zero delta, and re-run the reconstruction test to confirm the derived label is still accurate.
- • Delta between the source of truth and each derived store, per field, computed continuously.
- • Propagation lag per derived store against its declared acceptable staleness.
- • Rebuild exercises: last successful run, duration, and whether any field failed to reconstruct.
- • Write attempts against derived stores — should be zero, and each one is the start of the next divergence.
- • For externally owned state, the age of the last successful reconciliation against the external system.
- • Any system with more than one copy of the same information, which is any system with a cache or an index.
- • Incident response, where a named authority turns a debate into a mechanical repair.
- • Integrations with external systems, where being explicit about who is authoritative prevents the most common class of integration bug.
- • Collaborative multi-writer state, where a single authority is the wrong model and convergence is the right one.
- • When the label is asserted rather than tested, since an untested "derived" store is a liability disguised as a convenience.
- • When it is used to dismiss a real problem — "the owner is right by definition" is about repair direction, not about whether the owner’s value is correct.
- • Event log as the source of truth with all state as projections, which makes reconstruction routine and testable rather than exceptional.
- • Convergent replicated types for genuinely collaborative state, where no single authority exists and none is needed.
- • A single store with no derived copies: no authority question at all, at the cost of the read scaling and query flexibility the copies were for.
- • Explicit per-field authority split across systems, which is the honest model when an external provider owns part of your entity.
Delete everything but one store. Which one rebuilds the rest?
| Store | Derived from | After the test | What it is |
|---|---|---|---|
| append-only event log | nothing — it is written directly | gone forever | Every state change ever accepted, in order. Nothing derives it; everything derives from it. |
| orders-db | events | kept | Current state, folded from the log. Rebuildable by replay — but it holds no history the log does not. |
| search-index | orders or events | rebuildable | A read model shaped for one query. Purpose-built, fast, and entirely derived. |
| redis cache | orders or events | rebuildable | Copies of copies. Rebuildable from anything upstream, and stale by construction. |
| analytics warehouse | events | gone forever | Needs the history, not the current state — so the orders table cannot rebuild it. |
| partner's copy of our data | orders or events | rebuildable | Outside your blast radius and outside your control, but still a derived store you are responsible for repairing. |
| payments ledger | nothing — it is written directly | gone forever | Written directly by the payments provider's webhooks. Nothing in your system derives it — which makes it a second source of truth, whether or not anyone decided that. |
What people believe, and what is true
The database is the source of truth.
Only for fields it owns. With event sourcing the log is authoritative; for a subscription state your provider is; for some values a spreadsheet genuinely is.
When copies disagree, take the most recent one.
That hands the decision to clock skew, and a machine with a fast clock wins every time, permanently and silently.
The search index is derived, so we can drop and rebuild it any time.
Not if it holds fields nobody else has. Run the reconstruction test per field before treating any store as disposable.
We are the system of record for our customers.
For some fields. Identity may live with an IdP, payment state with a provider, marketing preferences with a platform. Being derived for fields on your own entity is normal and must be labelled.
The source of truth is right, so this value must be correct.
It is authoritative, not infallible. A bad write to the owner propagates everywhere with full confidence — which is why validation belongs at the owner.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The source of truth is the store you could rebuild everything else from. Name it per field. When copies disagree, it wins by definition.
Practical
Run the reconstruction test per field and record the answer where the data is. Label derived responses with freshness. Exercise rebuild paths on a schedule. Compare derived stores against the source continuously, since none of this drift produces errors.
Advanced
Naming a source of truth converts a consensus problem into a propagation problem. With several authorities you need agreement — locks, quorums, conflict resolution. With one, updates are one-way facts and every other store is a cache with a repair procedure. The engineering discipline is keeping that structure true as the system grows, because every convenient direct write and every accumulated field on a projection is a step back toward the consensus problem, taken by someone who did not realise they were taking it.
Apply it
- 🔧 Run the reconstruction test on your largest derived store, field by field. List every field that cannot be rebuilt.
- 🔧 Find one business-critical value in your organisation that is maintained outside any system of record. Decide who should own it.
- ⚡ A replay of the event log into a fresh projection produces different values from production. What does that tell you, and what is the sequence of repairs?
- 💬 How do you identify the source of truth for a piece of state?
- 💬 Two stores disagree and one has a newer timestamp. What do you do, and why is that not obvious?
- 💬 Your search index has a field that exists nowhere else. What does that make it, and what changes?
- 💬 A payment provider says the subscription lapsed; your database says active. Which is right, and what should your system have been doing?