Full Refresh vs Incremental
Rebuild everything every time, or process only what changed. The first is expensive and has no state to get wrong, and it is the right answer more often than people admit.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Should this model recompute all of history on every run, or only the rows that changed — and what does the second one oblige you to track forever?
The consumer never sees which was chosen and always feels it. Full refresh gives them a table that is internally consistent with its source on every run and gets slower every month; incremental gives them a fast pipeline and a table whose correctness depends on state they cannot inspect.
The choice is made per model, not per platform. The unit is one derived dataset and its rebuild strategy, and the same pipeline routinely contains both: dimensions refreshed fully, facts processed incrementally, because the volumes and the change patterns differ by orders of magnitude (Model Layering).
Rebuild everything, every night. CREATE OR REPLACE TABLE x AS SELECT ... FROM source. There is no watermark, no merge key, no late-data policy, no drift and nothing to reconcile: the output is a pure function of the input at the moment it ran. For most models at most companies this is correct, and it stays correct for far longer than the industry's enthusiasm for incrementality suggests.
The nightly rebuild stops fitting in the window. Not gradually — it fits, then it does not, and it does so on the night a marketing campaign doubles the day's volume (Compute Waste).
- The nightly rebuild stops fitting in the window. Not gradually — it fits, then it does not, and it does so on the night a marketing campaign doubles the day's volume (Compute Waste).
- The cost of the rebuild scales with all of history while the new information scales with one day, so the platform pays proportionally more to learn proportionally less every month.
- A full refresh re-reads the source at full width, which is fine against a lake and hostile against an operational database (OLTP vs OLAP).
- The refresh is not atomic — a
DROPfollowed by aCREATE— so a consumer querying during the rebuild sees an empty or partial table rather than yesterday's (Atomic Publish). - History that no longer exists in the source is silently discarded on every rebuild: a full refresh reproduces the source, including the rows the source has deleted, which is how a "safe" strategy quietly loses history (Keeping Raw History: The Recovery Position and the Liability).
- A model that references the current time is not reproducible under either strategy, and a full refresh makes that obvious every night by producing a different answer for the same historical period (Idempotent Data Pipelines).
What is actually happening
- A full refresh is a pure function of the source at read time. Nothing is remembered between runs, so there is no state to be stale, no watermark to be wrong and no late record to be missed — the output is always self-consistent with whatever the source said when it was read.
- An incremental model is a function of the source and of remembered state: a high-water mark, a set of processed partitions, or a merge key. Every failure mode unique to incremental processing is a failure of that state (The High-Water Mark).
- The state is what makes incremental fast and what makes it wrong. A watermark based on
updated_atmisses rows whose transaction committed after their timestamp was assigned; one based on a source log position does not, and requires a source that exposes one (Incremental Extraction). - Incremental models drift. Because they never re-read what they already processed, an error introduced once persists indefinitely, whereas a full refresh silently repairs yesterday's mistake tonight. This is the property most often forgotten and most consequential (Reconciliation).
- The two are not exclusive: the common mature pattern is incremental for the routine load with a periodic full refresh to bound drift, which buys most of the speed and caps how wrong the state can become (Incremental Processing).
- A full refresh is also a recovery strategy — it is the only one that needs no plan, because rebuilding from source is exactly what a correction would do anyway (Backfills).
What each strategy obliges you to get right
The comparison is usually framed as cost versus speed, which is the least interesting axis. The interesting one is state: what must the pipeline remember between runs, and what happens when that memory is wrong.
Read the "state to get wrong" column. A full refresh has an empty cell, and every entry in the incremental column is a real failure that has shipped in production somewhere this week. That is the whole argument, and it is why full refresh remains the correct answer for far more models than fashion suggests.
The last row is the compromise most mature platforms land on, and it is worth noting that it does not remove any of the incremental failure modes. It bounds how long each one can persist, which is a different and weaker property than not having them — and usually the right one to buy.
| Strategy | Per-run work | State to get wrong | A logic fix applies to | How it recovers |
|---|---|---|---|---|
| Full refresh | Proportional to all history, every run. | None. The output is a function of the source at read time and nothing else. | All of history, automatically, on the next run. | By running again. There is no separate recovery procedure, which is the property that makes it the right default. |
| Incremental by timestamp watermark | Proportional to what changed since the last high-water mark. | The watermark. A mutable updated_at misses rows that committed after their timestamp was assigned, permanently and silently (Incremental Extraction). | New rows only. History keeps the old logic until a backfill rewrites it. | Rewind the watermark and reprocess — safe only if the write is idempotent (Upserts and Merges). |
| Incremental by source log position | Proportional to changes read from the source's own log. | The stored offset, plus the source's retention: past it, the changes are gone rather than late (Offsets and Commits). | New rows only, same as above. | Rewind the offset within retention; beyond it, re-snapshot the source (Snapshot and Stream: the Bootstrap Problem). |
| Partition-level rebuild | Proportional to the partitions that could have changed. | Only the rule for which partitions are eligible — much smaller than a watermark and much easier to reason about (Partitioning). | Any partition you choose to rebuild, which makes a ranged correction natural. | Rebuild the partition. This is the middle option and it is under-used. |
| Incremental with a periodic full rebuild | Change-proportional most runs; history-proportional on the rebuild cadence. | All of the incremental state, with a bound on how long any error survives. | New rows immediately, all history at the next full rebuild. | Wait for the rebuild, or trigger it. The cadence is the honest statement of the correctness horizon. |
The column that decides most real arguments is the fourth. A team that fixes logic often and backfills rarely is a team that should think hard before going incremental.
Where the cost actually differs
The cost comparison people run is the scan: full refresh reads all of history, incremental reads a day, therefore incremental wins. That comparison is correct about the largest driver and omits three others, two of which are paid in engineering time and one of which is paid only when something goes wrong.
The weights below are relative and directional. The point is not their exact ratio; it is that the two cheapest-looking terms in the incremental column — the state machinery and the expected cost of getting it wrong — are real and are usually excluded from the decision entirely.
The practical consequence: for a model where the scan term is small, incremental is a net loss, because you are paying the bottom four drivers to save almost nothing on the top one. Establishing that the top driver is actually large is the whole of the analysis.
The term the comparison is usually about, and genuinely the largest one once history is big relative to a day. It grows every month whether or not anything changes (Scan Cost).
Watermark logic, late-data policy, merge key, overlap window, and the tests that keep them honest. Paid once and then again at every schema change.
Drift, a skipped range, a fix that never went retroactive — each ends in a backfill with its own validation and announcement (Backfills).
An idempotent write reads what it is updating. Real, recurring, and the price of re-runs being safe (Upserts and Merges).
The full-refresh cost divided by its cadence. Cutting this to save money is how a platform loses the only bound it had on incremental error.
Proportional to change rather than to history. The saving that motivates the whole exercise, and the only term that is genuinely small.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a model where history is large and daily change is small — the case that favours incremental most. Not measurements. Reverse the first and last weights for a small model and the ordering inverts entirely, which is the point.
Choosing, and choosing again later
The decision is per model and it should be revisited, in both directions. Models grow into incrementality; they also occasionally shrink out of it, when a retention policy or an archive removes the history that forced the choice.
The options below are ordered from least to most state. That ordering is deliberate: each step down the list buys performance with a new thing that can be wrong, and the honest way to make the decision is to require a named constraint before taking the next step.
The one thing that should not be a criterion is which strategy sounds more sophisticated. An incremental model with a watermark and a merge key protecting a table a full rebuild would reproduce in a minute is not a mature platform. It is four failure modes bought at retail.
What specific constraint prevents rebuilding this model from scratch on every run?
when The rebuild fits comfortably in its schedule, the scan is immaterial, and the source can serve a full read without harm.
cost Compute proportional to history, every run. Buys no state, retroactive fixes for free, and recovery by simply running again. This is the default and it should be argued away from, not into.
when The model is partitioned and only recent partitions can change.
cost One rule about which partitions are eligible, and nothing else. The best value in the list: most of the saving, almost none of the state (Partitioning).
when The source exposes a monotonic position (a WAL position, a binlog coordinate, a broker offset) and retains it long enough.
cost An offset to store and a retention window that bounds recovery. Correct where a timestamp watermark is not, and unavailable for sources that expose no such position (CDC vs Polling).
when The source only offers updated_at or equivalent, and the model cannot afford a full read.
cost The weakest option: rows whose transaction committed after their timestamp was assigned are missed permanently. Mitigate with an overlap window and an idempotent write, and reconcile regularly (Incremental Extraction).
when The model is incremental and someone needs to be able to state how wrong it can be.
cost The full-refresh cost amortised over the cadence. Bounds drift and makes logic fixes eventually retroactive, which is the property incremental otherwise gives up.
when A SaaS API with a short window, or an operational database that a full-width scan would damage.
cost Incremental is not a choice here, it is a constraint — and the raw layer becomes the only place a full rebuild can ever come from, which makes its retention a correctness decision (Keeping Raw History: The Recovery Position and the Liability).
How to build it
Most important first.
- Start with a full refresh and move to incremental when a specific constraint forces it — the run does not fit its window, or the cost is material, or the source cannot serve a full read. "It will not scale" is not one of those constraints until it does not.
- Make the full refresh atomic: build into a new relation and swap, so consumers never observe an empty or partially built table (Atomic Publish).
- If you go incremental, choose the watermark from a source-side monotonic value rather than a mutable timestamp, and accept that a source without one cannot be processed incrementally without loss (CDC vs Polling).
- Pair every incremental model with a periodic full rebuild whose cadence bounds the drift you are willing to accumulate — monthly for something reconciled monthly, and never is not a cadence.
- Make the incremental write a merge on the business key, so re-processing an overlapping range is safe and the watermark can be conservative rather than exact (Upserts and Merges).
- Overlap the incremental window deliberately: reprocess the last few periods every run so that late data and small watermark errors self-heal, and rely on idempotency to make the overlap free (Late-Arriving Data).
- Reconcile the incremental model against a full recomputation of a sample period on a schedule. That comparison is the only thing that detects drift, and it is the check almost nobody builds (Validating a Backfill Before You Publish).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A full refresh guarantees the output is consistent with the source as of a single read. It guarantees nothing about consistency with the *previous* run's output — a source that changed underneath produces a legitimately different answer for the same historical period.
- An incremental model guarantees only that everything its watermark saw has been processed. Everything the watermark did not see is absent, permanently and silently (The High-Water Mark).
- Neither guarantees atomicity of publication. That is a separate property of how the result is written, and a full refresh implemented as drop-and-recreate is the least atomic write in this module (Atomic Publish).
- An incremental model with a merge on the business key guarantees convergence for the rows it processes — a claim about the output write under an assumed unique key, and not a claim that everything that should have been processed was (Exactly-Once: Input Consumption, State Update, Output Write).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- For an incremental model, the check is a periodic full recomputation of a sample period compared row-for-row against what the incremental model holds. It is the only thing that detects drift, and it is worth its cost precisely because nothing else can see it.
- For a full refresh, the check is reconciliation against the source for a closed period, which is cheap because the model claims to reproduce the source anyway (Reconciliation).
- Both miss anything wrong in the source, and the incremental check misses drift in periods it did not sample — which makes the sampling strategy part of the check rather than an implementation detail.
- Incremental buys freshness by making the per-run work proportional to what changed instead of to what exists, which is what allows a model to run hourly rather than nightly (Incremental Processing).
- A full refresh gives coarser freshness with a stronger property: every run reflects the source in its entirety, so nothing accumulated from a previous run's mistake survives the night.
- The mixed strategy gives fresh data continuously and a bounded correctness horizon: whatever went wrong is repaired by the next full rebuild, and the cadence of that rebuild is the honest statement of how stale a correction can be (The Freshness SLO).
- A logic change under full refresh applies to all of history on the next run, automatically. Under incremental it applies only to new rows, so the model contains two logics separated by the deploy date, and nothing marks the boundary (Semantic Changes).
- That difference is the strongest under-appreciated argument for full refresh: it makes corrections retroactive for free, while incremental turns every logic change into a backfill (Backfills).
- A schema change is easier under full refresh — the new shape applies everywhere — and under incremental it produces a table whose older rows were written under the old shape, which is a compatibility problem inside a single table (Schema Evolution).
- Full refresh is its own recovery: whatever went wrong, the next run rebuilds from source. This is the single property that makes it the right default, and it is worth more than it looks on a cost chart.
- An incremental model recovers by rewinding its watermark and reprocessing the range with an idempotent write. That works exactly to the extent that the write is a merge and the source can still serve the range (Replay from the Log).
- A corrupted watermark is the worst case: the model believes it has processed data it has not, and nothing will re-read that range until a human resets the state by hand (The High-Water Mark).
What can go wrong
- A full refresh that no longer fits its window, discovered on the night volume spikes.
- A non-atomic full refresh leaving consumers reading an empty table mid-rebuild (Atomic Publish).
- An incremental watermark based on a mutable timestamp, permanently skipping rows that committed late (Incremental Extraction).
- Incremental drift accumulating for months, because nothing ever re-reads what was already processed.
- A logic fix applied to an incremental model and never backfilled, splitting the table into two definitions at the deploy date.
- A periodic full rebuild scheduled to bound drift, which is quietly disabled during a cost review because it looks like duplicated work — the mitigation removed for the reason it was expensive (Compute Waste).
- "Incremental is the mature choice." Incremental is the choice a constraint forces. Adopting it before the constraint exists buys nothing and takes on every failure mode in this module.
- "Full refresh does not scale." It scales until the run stops fitting its window or the scan cost becomes material, and for most models at most companies neither happens. Say which one you are hitting.
- "Our incremental model is correct because it reconciles." It reconciles for the period you checked. Drift is cumulative and unevenly distributed, and a model can reconcile beautifully in March while missing a category since January (Reconciliation).
- "We fixed the logic, and the model is incremental, so we are done." An incremental model applies the fix to new rows only. The table now holds two definitions with an invisible boundary at the deploy date (Backfills).
Operating it
- Runtime per run against the size of the window it processes. A full refresh whose runtime grows with history and an incremental model whose runtime grows with anything are both visible in one chart (Pipeline Metrics).
- The watermark value itself, plotted over time. A watermark that stalls, jumps or goes backwards is the incremental model's primary failure signal and is invisible in task status (The High-Water Mark).
- Rows processed per run versus rows changed at the source per run, whose divergence is drift becoming measurable (Volume Anomalies).
- Time since the last full rebuild, per incremental model, as a first-class freshness-style metric — it is the age of the correctness guarantee.
- At 10x history, full refresh of a large fact table stops being viable and the dimensions usually still refresh fully — which is why the decision is per model rather than per platform.
- At 100x, incremental is mandatory for facts and the interesting question becomes how to bound drift affordably: sampling, partition-level rebuilds on a rotation, or reconciliation against the source instead of a full recomputation.
- Small data at any scale of company should refresh fully. The most common over-engineering in this domain is an incremental model with a watermark, a merge key and a late-data policy protecting a table that a full rebuild would reproduce in the time it takes to read this sentence.
- Full refresh cost is proportional to all history, every run. That is the term everyone cites and it is genuinely the reason to move away, once history is large relative to a day (Scan Cost).
- Incremental cost is proportional to change, plus the merge's read of the target, plus the periodic full rebuild, plus the engineering time to build and maintain the state. The last term is the one omitted from every comparison and is often the largest (What Actually Drives Data Platform Cost).
- The cost of getting incremental wrong is a silent, permanent shortfall, discovered later and repaired by a backfill. That is a real expected cost and it belongs in the comparison.
- Partitioned storage changes the arithmetic: a "full refresh" of only the partitions that could have changed is a middle option with much of the simplicity and a fraction of the scan (Partition Pruning).
- Full refresh trades compute for the absence of state. It is more expensive per run and it has no watermark to corrupt, no drift to accumulate and no backfill required after a logic change.
- Incremental trades correctness surface for speed and cost. Everything it saves is real, and everything it introduces — watermarks, late data, drift, non-retroactive fixes — is also real and arrives later.
- The mixed strategy costs the sum of both implementations and is usually right at scale, because it buys the speed of one and the correctness horizon of the other.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe trade — recompute everything versus remember what you processed — exists in every transformation framework. What varies is how much of the state the tool manages for you, and no tool removes the drift that comes from never re-reading what was already processed.
- SCALE-SPECIFICBelow the point where a full rebuild fits comfortably in its schedule and its scan cost is immaterial, full refresh is simply better and incremental is a liability. Above it the advice inverts completely, and the boundary is a property of one model rather than of the platform.
- TOOL-SPECIFICdbt makes the choice a materialisation setting, so switching is a one-line change and
--full-refreshis always available as an escape hatch; a hand-rolled pipeline usually hard-codes the strategy in the load logic, which makes the periodic full rebuild something someone has to build rather than something they can run.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the analogous trade in delivery: rebuild the artefact from scratch, or patch what changed. The argument is the same one — reproducibility against speed — and the industry settled it there in favour of the full rebuild.