What Must Persist
Not everything the system knows must survive a restart. Orders must; a cart may; a product's "in stock" badge is recomputed. Sorting data into must-persist, may-persist and recompute is what keeps the schema small and the losses acceptable.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Of everything this system holds in memory at some moment, what must still be there after a restart, what may be lost, and what should be recomputed rather than stored?
I keep finding state in the app that I am not sure about. The cart is in the session; the "items in stock" badge is a column someone added; the order total is stored and also computed; and the read state in the chat app is in a Redis key someone thought was temporary. I do not know which of these are decisions and which are accidents.
Persist everything, to be safe. Every value the app computes gets a column, the session goes in the database, and nothing can ever be lost. It is the cautious choice and it removes the need to decide.
Storing a computed value creates a second source of truth. The stored total and the sum of line prices agree until a line changes, and then the store has two totals and no rule for which one is right (Source of Truth).
- Storing a computed value creates a second source of truth. The stored total and the sum of line prices agree until a line changes, and then the store has two totals and no rule for which one is right (Source of Truth).
- Persisting the ephemeral makes it permanent. A cart stored forever is a table of abandoned carts that has to be cleaned up, migrated, and reasoned about in every query that touches products.
- "Everything persists" says nothing about *what happens when it does not*. The real question is the failure: if this value is lost, who notices and what do they lose? Storing everything skips the question instead of answering it.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- For each piece of state, ask what is lost if this vanishes right now, and who would notice? If a person loses something they cannot recreate — an order they paid for, a message they sent, a file they uploaded — it must persist, durably, before they are told it succeeded. If they lose a few minutes of convenience — a cart, a draft, a filter — it may persist, and the decision is about cost and expectation. If nobody would notice because the value can be rebuilt from other data — a total, a count, a badge — it should not be stored at all, or only as a cache that admits it is one.
- Separate facts from derivations. A fact is something that happened and cannot be recomputed: this customer paid this amount at this time. A derivation is a function of facts: the total, the stock badge, the unread count. Persist facts; recompute derivations until a measured cost says to cache them — and then label the cache as one, with a rule for when it is stale (Denormalization on Purpose).
- For "may persist" state, decide from the user's expectation rather than from technical ease. A shopper expects the cart to survive closing the tab and probably expects it tomorrow; they do not expect it next year. That is a persisted cart with an expiry, and writing the expectation down is the decision.
- Persist the fact at the moment it becomes true, and acknowledge only after. The order is written before the confirmation page renders; the message is written before the sender sees the tick. Anything acknowledged from memory is a promise the system cannot keep across a restart (External Systems Fail).
The store's state, sorted by what is lost
The matrix is the store's running state with a loss sentence per row. The third column is the decision; the fourth is the part that makes it a decision rather than a feeling — the expiry, the staleness rule or the write-before-acknowledge check.
| State | If lost now… | Decision | What makes it a decision |
|---|---|---|---|
| Order, OrderItem, Payment | A customer paid and has no order | Must persist | Written in one transaction before the confirmation page |
| Product, stock | Nothing can be sold; admin work is gone | Must persist | Admin changes are rows, not memory |
| Cart | A shopper re-adds a few items | May persist | Survives a refresh and a day; expires after |
| Last-used address | A shopper retypes it | May persist | Stored on the customer; the order keeps its own snapshot |
| Order total | Nothing — recomputed from items | Recompute | The amount charged lives on Payment as a fact |
| "In stock" badge | Nothing — stock > 0 | Recompute | Never a column |
| Bestsellers list | Nothing — a query over OrderItems | Recompute, cached | Refreshed hourly; labelled as approximate |
| Session token | A shopper signs in again | May persist | Expiry is the security decision, not the persistence one |
Choosing per piece of state
The decision device is the same three questions in order. Most state answers the first question with "nobody" and the sorting ends there; the interesting rows are the ones that look like derivations and are facts.
This value is in memory. What happens to it?
when A person would lose something they cannot recreate — an order, a message, an upload, a read position.
cost A durable write on the happy path, a table to design, and a transaction boundary to get right.
when A person would lose convenience — a cart, a draft, a preference — and their expectation says it should survive a while.
cost Cleanup, and an argument about how long "a while" is.
when The value is a function of stored facts and nobody would notice its absence — a total, a count, a badge.
cost A query on every read; at some load this becomes the next option.
when The derivation is expensive and a measurement says so.
cost A second copy that must say how stale it may be, and code that respects the label.
The chat app's read state, sharpened
Read state is the row most teams get wrong, because it looks computable. The board shows the store's neighbour case: an unknown that was a feeling ("is read state a cache?") until it became a question with a concrete experiment.
- ✓Messages persist before the sender sees delivery.
- ✓Unread count is a derivation: messages after the read pointer.
- ✓Online status and typing indicators are never stored.
? Is read state just a cache?
becomes Can "Alice has read up to message N" be rebuilt from any other stored data if the Redis key is lost?
experiment Delete the key on a test instance and try to reconstruct it from messages and sessions; if it cannot be rebuilt, it is a fact and needs a row.
? Read pointer or per-message flags?
becomes Does anyone need to know that a specific message was read, or only how far the reader has got?
experiment List the screens that show read state; if all of them show "read up to here", a pointer per participant per conversation is the whole fact.
? How fresh?
becomes If the read pointer is written asynchronously and the writer restarts, what is the worst a user sees — a message marked unread that they read?
experiment Kill the writer between the UI update and the write; decide whether that outcome is acceptable, and if not, write before updating the UI.
How to do it
Most important first.
- List every piece of state the running system holds — in memory, in the session, in the browser, in a cache, in a queue — not only what is in the database.
- For each, write the loss sentence: "if this vanishes now, X loses Y". Sort by who X is and whether Y can be rebuilt.
- Mark facts and derivations. For every derivation you intend to store, write the rule that keeps it in step with its facts, or do not store it.
- For every "must", find the line of code that acknowledges success and check that the durable write happens before it (What Must Never Break).
- For every "may", write the expectation and the expiry, and put the expiry in the code.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- The store. Must persist: Product, Order, OrderItem (with price paid), Payment, the admin's stock changes. May persist: the cart (survives a refresh and a day; expires), the customer's last-used address (convenience). Recompute: order total (sum of items plus shipping — stored on the order only as the amount actually charged, which is a fact from Payment), "in stock" badge (stock > 0), "bestsellers" (a query over OrderItems, cached with a stated staleness).
- The chat app. Must persist: User, Conversation, Participant, Message — written before the sender sees delivery. Read state: a fact, not a derivation — "Alice read up to message N at time T" cannot be rebuilt — so it persists, and the "temporary" Redis key is promoted to a row. Recompute: unread count (messages after the read pointer), online status (lost on restart and nobody minds), typing indicator (never stored).
- The analytics dashboard. Must persist: the raw events, once, durably. Everything the dashboard shows is a derivation — daily totals, funnels, top products — and is recomputed or materialised with a stated refresh, because a "total" that cannot be rebuilt from events is a number nobody can check.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every piece of state has a loss sentence, and the "must" list is shorter than you expected.
- No stored value is a derivation without a written staleness rule beside it.
- Every acknowledgement of a "must" happens after a durable write, and you checked the line.
- The "may" state has expiries, and the abandoned-carts problem is a cron job, not a migration.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?If this value vanished right now, who would notice and what would they lose that cannot be rebuilt?
- ?Is this a fact — something that happened — or a derivation of facts?
- ?For every stored derivation, what is the rule that keeps it in step, and what happens when the rule is skipped?
- ?Where is the acknowledgement for this fact, and is the durable write before it?
What can go wrong
- Everything becomes "must". Persisting the typing indicator and the online status makes the database the bottleneck for a chat app before it has any users, and the loss sentence for each was "nobody notices".
- A derivation is stored "for performance" before any performance was measured, and the rule for keeping it in step is "we will remember to update it".
- A fact is misread as a derivation: read state, "last login", the price the customer paid — each looks computable from other data and is not, because the other data changes.
- Recomputing derivations costs queries on every read; at some scale a stored, labelled cache is right, and the loss sentence does not tell you when.
- Expiring "may" state means occasionally losing a cart a shopper wanted, and the expiry will be argued about.
- Writing before acknowledging makes the happy path slower by one durable write; it is the write that makes the acknowledgement true.
- "Persist means database." A durable queue, an object store or a write-ahead log all persist; the question is durability before acknowledgement, not which product.
- "Recompute means never store." A cache is a stored derivation with a staleness rule; the failure is a stored derivation that pretends to be a fact.
- "Sessions are ephemeral." The session *object* is; what it holds may not be — a cart in a session is a decision that the cart may be lost, and it should be a deliberate one.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALFact versus derivation and the loss sentence apply to any system with state, including a CLI tool with a config file and a pipeline with checkpoints.
- SCALE-SPECIFICAt small scale every derivation is recomputed and the schema is only facts. As reads grow, derivations get materialised one at a time, each with a measured reason — the move stays the same and the "recompute" column shrinks.
- ILLUSTRATIVEThe store, chat app and dashboard examples are invented; the day-long cart expiry is for the shape of the argument.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — Testing & Reliability is not a domain yet: the check that every "must" is written before it is acknowledged is a test worth writing by hand — kill the process between write and response and see what the user was told.