Kappa Architecture
One event log, one stream processing path, and reprocessing by replay. It removes Lambda's duplicated implementation and replaces it with two demands: the log must retain everything you might reprocess, and the stream job must replay history at a rate batch used to manage.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
If reprocessing is just replaying the same job from the start of the log, do you still need a batch layer — and what has to be true for that replay to be possible?
The same consumers Lambda served, minus the confusion about which layer produced their number. What they gain is stability of definition — one implementation, so one answer — and what they must accept is that a correction to history arrives at the speed of a replay (Who Actually Consumes This Data).
The unit is the event as retained in the log, and it is the pattern's single source of truth. Every output — a table, an aggregate, a materialised view — is a function of that log up to some offset, which is what makes a replay meaningful. If any input is not in the log, that part of the architecture is not Kappa, whatever the diagram says (The Event Log).
Delete the batch layer. You already have a stream job that computes the metric; when the logic changes, reset the offset to zero, let it recompute everything, and swap the output. One codebase, one engine, one mental model.
The log does not go back far enough. Retention was set as a cost decision at seven days, the bug has been present for four months, and the events needed to correct it no longer exist anywhere the job can read (Retention and Replay).
- The log does not go back far enough. Retention was set as a cost decision at seven days, the bug has been present for four months, and the events needed to correct it no longer exist anywhere the job can read (Retention and Replay).
- The replay cannot keep up. Reprocessing a year of events through a job sized for the live arrival rate takes far longer than a batch job that could partition the same year and process it in parallel (Distributed Data Processing).
- The state does not fit. A stream job that holds per-entity state for a rolling window is comfortable live and enormous when replaying a year, because the replay compresses a year of key arrivals into hours (Streaming State).
- Some inputs are not events. A vendor drops a CSV weekly, a SaaS API is polled, an analyst maintains a mapping table. None of those is in the log, so the "one path" now has a batch path attached to it that nobody counted (Ingestion Sources).
- The new job version cannot start from the old job's checkpoints, because its state schema changed. Replay from zero is the only option, which is exactly the expensive operation you were hoping to make routine (Checkpointing).
- Watermarks behave differently under replay. Events that were late in real time arrive within milliseconds of their predecessors during a replay, so windows that dropped them live now include them — and the replayed output legitimately differs from the original (Watermarks).
What is actually happening
- Kappa's claim is narrow and correct: if every input is an event in a retained log, and the processing is a deterministic function of that log, then reprocessing is the same program run from an earlier offset. There is no second implementation because there is no second execution model (Deterministic Replay: Making the Schedule Reproducible).
- Reprocessing is done by running a new instance of the job from offset zero (or from a chosen offset) writing to a new output, letting it catch up to the live position, and then switching readers to it. The old output serves throughout, which makes the operation safe (Atomic Publish).
- That construction is genuinely simpler than Lambda in one specific way and no other: there is one rule, in one place, so the divergence failure Lambda is defined by cannot occur. Everything else it demands is new.
- The first demand is retention as a correctness property. The log is the master dataset, so the reprocessing horizon and the retention horizon are the same number. Retention stops being a storage argument and becomes "how far back can we fix a bug" (Replay from the Log).
- The second demand is replay throughput. A batch layer processes a year by partitioning it across many workers with no ordering obligation between partitions. A stream job replaying the same year must respect its own state and ordering semantics while consuming far faster than it was sized for, and the parallelism available is bounded by the partition count of the topic (Topics and Partitions).
- The third demand is that replay is not automatically identical to the original run. Anything time-dependent — a watermark, an allowed-lateness policy, a call to an external service, a lookup against a mutable dimension — makes the replayed result differ from what was originally emitted, and that difference is usually discovered by a consumer (Processing Time).
Reprocessing as a deployment, not as a second system
The whole pattern rests on one substitution. Lambda answers "how do we correct history" with a batch layer that recomputes it. Kappa answers the same question with the same job, started earlier. If that substitution holds, the second implementation is unnecessary and everything Lambda spends on keeping two rules in agreement is saved.
Operationally it looks like a deployment rather than a repair. Start a new instance of the job from the offset you want, point it at a new output, let it consume forward until it reaches the live position, verify, then switch readers. The old output serves the whole time, which makes this one of the few genuinely safe corrective operations in this domain (Atomic Publish).
The diagram makes the two failure surfaces visible. Everything to the left of the log must actually be *in* the log — a source that is polled or dropped as a file is a second path, and the pattern's claim does not cover it. Everything to the right must be deterministic, or the replay is a new computation rather than a reproduction of an old one.
What replay actually demands
Kappa is often summarised as "simpler than Lambda", which is true of exactly one thing and misleading about everything else. It removes the duplicated implementation. In exchange it makes two demands that a batch layer never made, and both of them are silent until the day you depend on them.
The first is that the log retains everything you might reprocess. A batch layer reads a master dataset held in storage designed to be cheap and permanent; a Kappa platform reads a log with a retention setting, and the person who set that number was usually thinking about broker disk rather than about how old a bug might be when it is found (Retention and Replay).
The second is throughput. A batch job that recomputes a year does so by splitting the year across many workers with no ordering constraints between them. A stream job replaying the same year is constrained by its own state, its ordering semantics and the partition count of the topic, and must consume vastly faster than the rate it was sized for. It is entirely normal for a replay to be slower than the batch backfill it replaced, and to discover that during an incident (Backfills).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A logic bug is found in four months of history. | Reprocessing cannot start; the events are no longer retained. | Retention was set as a storage decision, not as a statement of how far back corrections must be possible. | Re-derive retention from the reprocessing requirement and use tiered log storage for the long tail. Meanwhile, recover by snapshotting source state and stitching the stream onto it (Snapshot and Stream: the Bootstrap Problem). |
| A full replay is started. | Catch-up is projected in days; the correction is needed today. | Replay parallelism is bounded by topic partitions and by the job's state and ordering semantics, while a batch job over the same range had no such bound (Distributed Data Processing). | Measure replay throughput on a schedule so the number is known before it is needed, and provision a temporary higher-parallelism job for reprocessing where the partition count allows. |
| A replay of long history is running. | The job fails on state size, or the state backend degrades steadily. | Replay compresses a year of key arrivals into hours, so far more keys are live in state concurrently than during normal operation (Streaming State). | Test replays over the full horizon rather than over recent windows, and size the reprocessing configuration separately from the live one. |
| Replayed output is compared with the original. | The numbers differ for periods where no logic changed. | Non-determinism: wall-clock branching, allowed-lateness evaluated against processing time, or a lookup against a reference table that has since changed (Processing Time). | Version reference data into the stream, express lateness against event time, and add a replay-equivalence test over a fixed historical period to the build (Determinism: Same Input, Same Output?). |
| A small logic change is deployed. | The new job cannot restore the old checkpoints and must start from zero. | The state schema changed, and checkpoint compatibility across job versions is narrower than most teams assume (Checkpointing). | Treat state migration as a first-class deployment concern: version the state, run the new job alongside the old, and swap on catch-up. |
| A vendor file arrives weekly and a SaaS API is polled nightly. | The "single path" has a batch job beside it that nobody monitors or documents. | Not all inputs are events, so the pattern's precondition never held (Ingestion Sources). | Either land those sources into the log as events so replay covers them, or document the batch path honestly as part of the architecture rather than as an exception. |
When it is genuinely the simpler answer
The honest comparison is not Kappa against Lambda in the abstract. It is a set of conditions, and the pattern is straightforwardly correct when they hold and straightforwardly expensive when they do not.
It holds best when the source is already a log — change data capture from operational databases, clickstream, telemetry — the transformations are expressible as stream operations, history is short enough that a full replay is a routine operation, and the team has genuine stream-processing expertise. Under those conditions Kappa is less machinery than any alternative, and the single-definition property is a real ongoing benefit.
It holds worst when history is long, when several inputs are files or API pulls, when the transformations are large joins over historical ranges that a stream engine expresses awkwardly, or when the team has one stream engineer. In those cases the pattern imposes a permanent retention bill and an unpractised recovery path in exchange for removing a duplication that a shared dialect might have removed anyway (Batch and Streaming Unification).
Paid continuously and proportional to how far back you promise to be able to correct. The defining cost of the pattern and the one that is set by a recovery decision rather than a storage one.
Bursty and large: reads the whole retained horizon and rewrites the whole output. Its magnitude is the same whether it runs in hours or days; only the concurrency and the wait differ.
Held whether or not events flow, which makes low-volume pipelines disproportionately expensive compared with a scheduled job doing identical work.
Grows with key cardinality and window length, and grows sharply during replay because arrivals are compressed in time.
Lower than Lambda by exactly one implementation and higher than a batch platform by the event-time and state expertise required. The comparison people make on this line is usually against Lambda and rarely against a plain scheduled platform.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative and unitless, to establish an ordering. The teaching is the top two rows: this pattern converts a maintenance cost that Lambda paid in engineering time into an infrastructure cost paid in retained bytes and replay compute. That is often a good trade and it is never a free one.
Check the preconditions in order — the first failure is your answer.
when Every source publishes to the log; nothing arrives as a file drop or an API pull.
cost If false, you have a batch path regardless. The choice is whether to document it or pretend the diagram is accurate (Ingestion Sources).
when The log holds events for at least as long as a bug might plausibly go unnoticed.
cost Retained bytes for the full horizon, paid continuously. This is the pattern's signature cost and it buys exactly the recovery guarantee (Retention and Replay).
when Reprocessing the whole horizon completes within the window in which a correction is still useful.
cost Provisioning replay parallelism, testing it regularly, and accepting that partition count bounds it. If a replay takes days, the recovery mechanism is nominal (Backfills).
when No wall-clock branching, no unversioned reference lookups, no non-idempotent external effects.
cost Discipline that ordinary stream jobs do not observe, plus a replay-equivalence test in the build (Determinism: Same Input, Same Output?).
when Event time, watermarks, state backends and checkpoint migration are understood by more than one person.
cost The expertise is the real barrier. A platform whose recovery path only one engineer can execute has a single point of failure with a pulse (Watermarks).
when Any of the above is false and cannot cheaply be made true.
cost Keep a batch path — either as Lambda with a disciplined shared definition, or as a plain scheduled platform over immutable files. Both are legitimate, and both are cheaper than a replay guarantee that does not work (Lambda Architecture).
How to build it
Most important first.
- Verify the precondition before adopting the pattern: is every input genuinely a retained event, or are some of them file drops and API pulls? If the honest answer is the latter, you will have a batch path regardless, and the choice is whether to acknowledge it (Batch vs Streaming Ingestion).
- Set retention from the reprocessing requirement, not from a storage budget. Write the sentence "we can correct a bug up to N days old" and make retention satisfy it; tiered or archived log storage exists precisely so that horizon is not capped by broker disk (Storage Lifecycle).
- Make the job deterministic with respect to the log. No wall-clock branching, no reads of mutable reference data without versioning it into the stream, no non-idempotent external effects. Determinism is what makes a replay a recomputation rather than a new experiment (Determinism: Same Input, Same Output?).
- Always replay into a new output and swap, never in place. A job replaying into the live table publishes a partial history that never existed, and any consumer reading during the catch-up gets it (Atomic Publish).
- Size and test the replay path deliberately: run a full reprocess on a schedule, in a lower environment or as a shadow output, so that the operation you depend on for recovery is one you have actually performed recently (Validating a Backfill Before You Publish).
- Plan for state migration explicitly. A job whose state schema changed cannot resume from old checkpoints, so version the state, keep the old job running until the new one has caught up, and treat this as the standard deployment path rather than an exception (Rolling Back Data).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Reproducibility of the output from the log, within retention and given determinism. Both conditions are load-bearing; without either, the pattern's central promise is not available (Reprocessing vs Retrying).
- One implementation, therefore one definition. This is the guarantee Lambda cannot give and the strongest argument for the pattern (The Metrics Layer).
- Per-partition ordering only, inherited from the log. A stream job cannot restore a global order the log never had (CDC Ordering and Transaction Boundaries).
- Effectively-once processing is available per pipeline and must be constructed: it requires the state update and the output write to be transactional together, or the output write to be idempotent. It is not a property the pattern confers (Exactly-Once: Input Consumption, State Update, Output Write).
- No guarantee that a replay produces byte-identical output to the original run. Time-dependent windowing, lateness policies and external lookups all break that, and the difference is legitimate rather than a bug (Late Events).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check this pattern needs is a replay equivalence test: take a completed historical period, reprocess it into a scratch output, and compare against what the live job produced for the same period. Differences are either a genuine bug fix or an unnoticed non-determinism (Reconciliation).
- Add a retention headroom check: the age of the oldest retained event compared with the reprocessing horizon you promised. When headroom reaches zero, the pattern's recovery guarantee has silently expired and nothing will report it (Retention and Replay).
- Both miss incompleteness of the log itself. If a capture gap means an hour of changes was never published, every replay reproduces the same gap faithfully. Only a reconciliation against the originating system finds that (CDC Failure Modes and the Retention Deadline).
- Live freshness is the pattern's strong point: output lags the log by the job's processing time, with no scheduling interval added.
- Correction freshness is the pattern's weak point and the one that surprises people. A bug found today in six months of history is fixed at the speed of a replay of six months, which may be considerably slower than a batch backfill of the same range (Backfills).
- During a replay the new output is maximally stale by design — it starts at the beginning of history and works forward. This is why the swap-on-catch-up construction matters: consumers must not see the rebuilding output until it has reached the live position (Atomic Publish).
- Changing the job means changing one implementation, which is the pattern's point. Deploying that change usually means a replay, which is the pattern's price — so the frequency of logic changes and the cost of a replay multiply together into the real maintenance load.
- Old events in the log were written under old schemas, so a replay reads every historical version. Forward and backward compatibility are not optional here; they are the condition under which reprocessing works at all (Forward Compatibility).
- State schema evolution is the hardest case: a new job version generally cannot restore checkpoints written by the old one, so an apparently small change becomes a full reprocess (Checkpointing).
- Semantic change in the events themselves is invisible to a replay, which will faithfully reprocess a field that meant one thing in March and another in June with a single rule (Semantic Changes).
- The recovery story is the pattern: reset to an offset, reprocess into a new output, verify, swap. When the preconditions hold this is genuinely excellent — no coordination with producers, no load on source systems, one procedure for every kind of correction (Replay from the Log).
- It fails absolutely at the retention boundary. Beyond it, recovery requires a snapshot of source state plus the stream stitched onto it, which is the same construction CDC uses on first run and is considerably more work than a replay (Snapshot and Stream: the Bootstrap Problem).
- Partial recovery — reprocessing one key range or one time range rather than everything — is harder in a stream job than in a batch one, because offsets are positional rather than predicate-based. Designing outputs so that a range can be replaced atomically is what makes targeted recovery possible (Planning a Backfill).
- Keep the previous output alive until the replacement has been validated. The swap is the only reversible step in the whole procedure (Rolling Back Data).
What can go wrong
- The retention horizon quietly falls short of the reprocessing requirement, and it is discovered during the incident that needed it.
- A replay of long history saturates the job, the state store or the sink, and takes long enough that the correction is delivered days after the question was asked.
- A non-deterministic element — a wall-clock branch, a lookup against a mutable table — makes replayed output differ from original output, and the difference is read as data corruption.
- Inputs that are not events get quietly handled by a side batch job, so the platform is Lambda with one layer undocumented (Data Platform Anti-Patterns).
- A state-schema change forces a full reprocess for a small logic fix, converting a routine deploy into a multi-day operation.
- The mitigation failing: replays are tested only on short recent windows, so the throughput, state size and schema-version problems that only appear over long horizons are never exercised until the day they matter (Validating a Backfill Before You Publish).
- "Kappa is the simpler architecture." It is simpler in one dimension — one implementation — and adds two demands that Lambda did not make. Neither pattern is simpler in general; they are simpler under different conditions (Lambda Architecture).
- "Replay means we can always reprocess." Replay reaches exactly as far back as retention, and no further. The horizon is a number someone chose, usually for cost reasons, usually before anyone framed it as a recovery window.
- "We removed the batch layer." Check whether every input is genuinely in the log. A weekly vendor file and a polled API mean the batch layer still exists, undocumented and unmonitored (Ingestion Sources).
- "A replay reproduces the original output." Only if the job is deterministic with respect to the log. Time-dependent windowing and external lookups make the replay legitimately different, and a consumer will read that difference as an error (Determinism: Same Input, Same Output?).
- "Streaming is more modern, so this is the direction of travel." Compare freshness, complexity, cost, failure handling and operational burden. A nightly batch job over immutable files is reproducible, cheap and easy to reason about, and for a great many datasets it remains the correct answer (Batch vs Streaming Ingestion).
- A long retention horizon is a large standing collection of personal data. The recovery argument for retaining a year and the privacy argument for retaining a week are both legitimate, and this pattern forces the organisation to settle them explicitly per topic (Data Retention).
- Deletion requests are structurally awkward against an append-only log that the architecture depends on being complete. Keeping identifiers out of the log, or tokenising them so the token map can be deleted, are the constructions that keep both the replay guarantee and the deletion obligation intact (Deletion Requests, Data Masking, Tokenisation & Encryption).
Operating it
- Retention headroom — the age of the oldest retained event minus the promised reprocessing horizon — published as a single number per topic. It is the pattern's most important and least-monitored signal (Retention and Replay).
- Replay throughput measured as event-time progress per unit of wall-clock time during a reprocess, which tells you how long a full rebuild would take before you need one (Pipeline Metrics).
- State store size and growth per job, both live and during replay. The replay figure is the one that predicts failure (Streaming State).
- Time since the last successful full reprocess in a non-production environment. A recovery mechanism that has not been exercised is a claim, not a capability.
- Divergence between replayed and original output for a fixed historical period, which is the direct measure of the job's determinism (Quality Alerting).
- At 10x event volume, live processing scales with partitions and parallelism; the replay path scales worse, because the same partition count now has ten times as much history to pull through it.
- At 10x history length, retention cost grows linearly and replay time grows linearly, and the second one is what actually removes the pattern's advantage — a recovery mechanism too slow to use is not a recovery mechanism.
- At 10x state cardinality, the replay is the binding constraint long before live processing is, because replay compresses arrivals in time and therefore holds more keys concurrently (Streaming State).
- Below a modest event rate the pattern is mostly cost with little benefit: a scheduled job over retained files gives the same reproducibility with none of the state or watermark reasoning (Batch vs Streaming Ingestion).
- Retained bytes, driven directly by the reprocessing horizon. This is the pattern's signature cost: the recovery guarantee is literally paid for in storage, continuously, whether or not it is used (What Actually Drives Data Platform Cost).
- Replay compute is bursty and large — a full reprocess reads all retained history and writes all output, concentrated into the time you are willing to wait (Reprocessing vs Retrying).
- Continuous stream compute is paid at all times, including when event volume is near zero, which makes low-volume Kappa pipelines expensive relative to a scheduled job doing the same work (Cost vs Freshness).
- Engineering cost is lower than Lambda's by exactly one implementation, and higher than a batch platform's by the expertise required to reason about event time, state and watermarks (Cost Attribution).
- You trade a duplicated implementation for a retention obligation and a replay-throughput obligation. That is a good trade when your inputs are a log and your history is short, and a poor one when history is long and some inputs are files.
- One codebase means one definition and also one blast radius: a bug in the single implementation is wrong everywhere, with no independently-computed second view to disagree with it.
- Determinism is what makes the pattern work and it forbids a set of convenient things — wall-clock logic, un-versioned reference lookups, non-idempotent side effects — that ordinary stream jobs do routinely.
CDC retention deadline
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe core claim — that reprocessing is the same program from an earlier offset when every input is a retained event and the job is deterministic — is a property of log-based computation and holds across products. So do its two preconditions.
- BROKER-SPECIFICHow far back you can replay and what it costs differ sharply: some brokers cap practical retention at local disk while others tier older segments to object storage, and consumer position management differs between offset-based and acknowledgement-based systems, which changes how a targeted replay is even expressed.
- ENGINE-SPECIFICReplay throughput, state backend size limits, checkpoint compatibility across job versions and watermark behaviour under accelerated consumption are all engine properties. Two stream processors given the same log and the same logic can differ by more in replay behaviour than in live behaviour.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what a checkpoint restores, why a replay of a distributed stateful computation is not automatically deterministic, and what ordering a partitioned log can and cannot give back.
- — DevOps / Production Engineering owns the deployment shape this pattern depends on: running two versions of a job side by side, migrating state across versions, and swapping consumers onto a rebuilt output without a flag day.