Dataevent sourcingevent logreplaysnapshotprojection

Event Sourcing

Instead of storing balance = 100, store AccountCreated +100, PaymentMade −20, RefundReceived +20 and derive the balance by replaying the log; you gain a complete audit trail, time travel and rebuildable projections, and pay with snapshots, event versioning, GDPR pain and a model most applications do not need.

▶ InteractiveInterview questionDebug it
Progress
What problem does this solve?

A row that stores current state destroys history: after UPDATE accounts SET balance = 100 nobody can answer "why is it 100?", "what was it on Tuesday?" or "rebuild the read model from scratch". Event sourcing stores the facts that happened, in order, as the source of truth, and treats every state as a derivation of them.

State as a fold over events

A conventional system stores accounts(id, balance) and overwrites the balance on every change. An event-sourced system stores an append-only log per aggregate: AccountCreated { initial: 100 }, PaymentMade { amount: 20 }, RefundReceived { amount: 20 }. The current balance is not stored anywhere authoritative; it is computed by replaying the events through a reducer — 100 − 20 + 20 = 100. The log is the truth; everything else is a cache of it. Writes append an event after validating the command against the replayed state; they never UPDATE.

Replay is a fold, exactly like a running total: state after event *n* is apply(state after n−1, event n). That is also why snapshots work — store the state at event 1,000 and replay only events 1,001 onward, the same trick as a Prefix Sum cached at checkpoints. Without snapshots an aggregate with a million events takes a million steps to load; with a snapshot every 100 events it takes at most 100.

Replay is a reduce; a snapshot is a cached prefix
1type Event =
2 | { type: 'AccountCreated'; initial: number }
3 | { type: 'PaymentMade'; amount: number }
4 | { type: 'RefundReceived'; amount: number }
5
6const apply = (balance: number, e: Event): number => {
7 switch (e.type) {
8 case 'AccountCreated': return e.initial
9 case 'PaymentMade': return balance - e.amount
10 case 'RefundReceived': return balance + e.amount
11 }
12}
13
14async function load(accountId: string) {
15 const snap = await snapshots.latest(accountId) // { seq: 1000, balance: 340 } or null
16 const events = await log.after(accountId, snap?.seq ?? 0) // only the tail
17 return events.reduce(apply, snap?.balance ?? 0)
18}
19
20async function pay(accountId: string, amount: number, expectedSeq: number) {
21 const balance = await load(accountId)
22 if (balance < amount) throw new Error('insufficient funds')
23 await log.append(accountId, { type: 'PaymentMade', amount }, expectedSeq) // optimistic concurrency on seq
24}
The same account, two ways
state storage                       event sourcing
accounts                            account_events (append-only)
id    balance                       seq  type             data           at
acc1  100                           1    AccountCreated   {initial:100}  09:00
                                    2    PaymentMade      {amount:20}    09:05
                                    3    RefundReceived   {amount:20}    09:40
"why 100?"  → unknown               "why 100?"  → replay: 100 − 20 + 20
"at 09:10?" → unknown               "at 09:10?" → replay events with at ≤ 09:10 → 80

What the log gives you

Auditability for free: every change is a first-class record with a timestamp, a cause and (if you store it) an actor — regulators and support teams ask "what happened to this account?" and the log *is* the answer. Temporal queries: the state at any past moment is a replay up to that moment. Rebuildable projections: every read model (CQRS) is derived, so a projector bug is fixed by correcting the code and replaying the log into a fresh store; a new screen is a new projector run over history. Debugging by replay: copy the events of one aggregate into a test and reproduce the exact sequence that led to the bad state. Integration: the events the system needs internally are the events other services want (Event-Driven Architecture), already ordered per aggregate.

The relationship to a Kafka-style log (Kafka-Style Logs: Topics, Partitions, Offsets) is close but not identity. A partitioned log gives ordered, replayable, retained events per key — an excellent transport and a plausible event store for some systems — but an event store also needs per-aggregate reads (all events for acc1, cheaply), optimistic concurrency on append (expectedSeq), and indefinite retention; a database table with (aggregate_id, seq) as the primary key is the more common event store, with the log used for publishing.

Command → events → state and projections
Pay 20events after snapshotlatest snapshotappend PaymentMadebalance, statements, …ClientSnapshotsAggregate (replay + validate)Event store (append-only)Event busProjectorsRead models
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

What it costs

Event versioning: events are forever, and the code that reads them changes. PaymentMade { amount } becomes PaymentMade { amount, currency } and every reducer must handle both — upcasting (transforming old event shapes into the current shape on read) is a permanent layer of the system, and a badly designed early event is a tax paid on every replay for years. Snapshots are mandatory at scale and are a cache with its own invalidation problem when the reducer changes. Queries against the log are impossible — "all accounts with balance under zero" requires a projection, which is why event sourcing drags CQRS in with it. Eventual consistency between the log and every read model is the normal state.

Deletion is the hardest. GDPR erasure requests demand removing personal data, and an immutable log is built to make removal impossible. The workable answers — crypto-shredding (encrypt each person's data with a per-person key and delete the key), storing PII outside the events by reference, or rewriting the log — are all real engineering, and none is as simple as DELETE FROM users. Finally, the mental model is unfamiliar: most developers, ORMs, admin tools and reporting systems assume current-state rows, and every one of them needs a projection to work.

Event sourcing vs conventional state persistence
State persistenceEvent sourcing
Source of truthCurrent rowsAppend-only event log
HistoryLost on update (unless audit tables)Complete, by construction
"What was it on Tuesday?"Not answerableReplay up to Tuesday
ReadsDirect query, joins, indexesOnly via projections (CQRS required)
WriteUPDATE rowAppend event with optimistic concurrency on seq
Load an aggregateOne rowSnapshot + replay tail
Schema changeALTER TABLE, migrate rowsUpcasters forever; events are immutable
Delete personal dataDELETECrypto-shredding or log rewrite
Rebuild a read modelRe-query the tablesReplay the log — always possible
Tooling and familiarityUniversalSpecialised; every tool needs a projection

Who actually needs it

Event sourcing is the right model when the history is the product: ledgers and accounting (a balance without its postings is meaningless), trading and order books, version-controlled documents, workflow engines, anything where "show me exactly what happened and why" is a core requirement rather than a nice-to-have. In those domains the events already exist in the business language — a posting, a fill, an edit — and storing them is more natural than storing the state. It also pays where audit trails would otherwise be built anyway, and in Saga Pattern orchestrators, whose state is naturally a sequence of step outcomes.

It is the wrong model for a CRUD application with a profile page, a settings form and a list of things. A conventional table with an updated_at and, where required, an audit table written by a trigger gives 90% of the audit value at 5% of the cost. The honest position: most applications need neither CQRS nor event sourcing; adopt event sourcing per aggregate where history is genuinely required, keep the rest as state, and do not rewrite a working system to it because the pattern is elegant.

Key points

  • Store events, derive state: the log is the truth and every current-state view is a rebuildable cache of it.
  • Replay is a fold; snapshots are cached prefixes so loading an aggregate stays bounded.
  • You gain audit, temporal queries, rebuildable projections and replay-driven debugging.
  • You pay with upcasting forever, mandatory snapshots and projections (CQRS), eventual consistency, and GDPR deletion via crypto-shredding.
  • Use it where history is the product — ledgers, order books, workflows; most applications need neither CQRS nor event sourcing.

Event log → replay → state

Event log → replay → state
An account stored as events. Append some, then replay them to rebuild the balance — with snapshots, or by time-travelling to an earlier point.
Append an event
Storage model
#1AccountCreated +100 → balance 100
#2PaymentMade -20 → balance 80
#3RefundReceived +20 → balance 100
#4FeeCharged -5 → balance 95
events
4
replay cost
4 events read
snapshots
0
balance
0 (replaying…)
Replay folds 4 events from zero to reach 95. Cost grows with the log: at 4 events it is trivial, at a million events per account it is not — which is why snapshots exist. Toggle them on and watch the replay cost drop. The costs on the other side: every event type is a versioned contract forever (old events must still replay), and deleting data — a GDPR request — means rewriting or crypto-shredding an immutable log.
1/5 · startbalance so far: 0

How data moves through it

One request or event, hop by hop.

  1. 1Client → Aggregate service: Pay(acc1, 20).
  2. 2Aggregate → Snapshot store + Event store: load the latest snapshot and the events after it; replay to the current balance.
  3. 3Aggregate → Event store: validate, then append(acc1, PaymentMade{20}, expectedSeq=3) — conflict if another writer got there first.
  4. 4Event store → Event bus → Projectors: PaymentMade fans out; the balance projection upserts acc1: 80, the statement projection appends a line.
  5. 5Client → Read model: GET /accounts/acc1 reads the projection (80), possibly a few milliseconds behind the log.

When to use — and when not

Use it when
  • The domain is a ledger: balances, positions, inventory movements, where the postings are the business and the total is derived.
  • Regulatory or product requirements demand a complete, tamper-evident history of every change with its cause.
  • Read models must be rebuildable from scratch and new projections added over history — analytics, ML features, new screens.
  • Workflows and sagas whose state is naturally the sequence of what happened.
Avoid it when
  • CRUD domains where current state is all anyone asks for; an audit table covers the rest.
  • Heavy personal data with strict erasure obligations and no appetite for crypto-shredding.
  • A team unfamiliar with the model and without time to build snapshots, upcasters, projections and rebuild tooling before shipping.
  • Retrofitting a working state-based system for elegance rather than a measured need.

Tradeoffs

Complexity
low → high
Ops cost
low → high
Latency
low → high
Consistency
weak → strong
Scalability
poor → strong

The highest-complexity pattern in this module: everything is derivable and auditable, but every read is a projection, every schema change is an upcaster, and deletion is a research project.

How it fails

  • No snapshots: an aggregate with 2 million events takes seconds to load and the write path times out.
  • An old event shape with no upcaster: the reducer throws on replay and the aggregate cannot be loaded at all.
  • Two commands append concurrently without optimistic concurrency on seq: both validate against the same balance and overdraw the account.
  • Projections treated as authoritative: a command validates against a lagging read model and accepts an invalid state change.
  • Personal data written into events with no erasure strategy; the first GDPR request has no technical answer.
  • Events that record *decisions* instead of *facts* (BalanceSetTo 80 rather than PaymentMade 20) lose the meaning that made the log valuable.

How it scales

  • Append-only writes are cheap and sequential; the store partitions cleanly by aggregate id.
  • Snapshots bound load time per aggregate; take one every N events or on a size threshold.
  • Projections scale as in CQRS — many projectors, each keyed by aggregate id for ordering, scaled on lag.
  • Log size grows forever; archive cold aggregates to object storage and keep snapshots hot.

How it interacts with databases, queues, caches, APIs and external systems

  • Database (event store): a table keyed (aggregate_id, seq) with a unique constraint providing optimistic concurrency; append-only by policy and by permissions.
  • Storage: snapshots and archived cold streams in object storage.
  • Queue/log: publishes appended events to projectors and other services; a Kafka-style log doubles as the transport but rarely as the store.
  • Cache/document store: read models — every query the system answers is served from one of these.
  • External systems: never replayed into — replaying a log must not re-send emails or re-charge cards; projectors that call out must be guarded by a replay flag and idempotency keys.