Lazy Evaluation
Transformations build a plan; nothing runs until an action asks for a result. That is what lets the optimiser see the whole query — and why your error message points at the wrong line and your pipeline ran three times.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Why did the error appear at write() when the mistake is twenty lines above it, and why did adding a debug count() double the job's runtime?
Anyone debugging a distributed transformation, and the cost line of any pipeline where the same chain is consumed more than once. Laziness is invisible when everything works and is the explanation for most of the confusing cases when it does not.
The unit is the plan: a graph of declared transformations, not the data itself. Until an action is called, nothing in the program has touched a row, and the variable holding your "DataFrame" holds a description of how to produce one.
Read the code as a sequence of steps that each produce data: read the file, filter it, join it, and now you have a result you can inspect. It is the natural reading, it matches how a single-machine script behaves, and it is wrong in a way that only shows up under specific conditions.
A type error in a transformation surfaces at the action, so the stack trace names the write and not the cast that caused it. On a long chain this turns a two-minute fix into an afternoon.
- A type error in a transformation surfaces at the action, so the stack trace names the
writeand not the cast that caused it. On a long chain this turns a two-minute fix into an afternoon. - The same chain is consumed by two actions — a count for logging and a write for the output — and the entire computation, including the source scan and every shuffle, happens twice (Compute Waste).
- A debug
count()added temporarily to check progress becomes permanent, and every run since has silently done the work twice. - The chain reads a source that changes between the two executions — a table being written, a
now()in a predicate — so the two runs of the same plan disagree, and the output is internally inconsistent (Determinism: Same Input, Same Output?). - Someone caches an intermediate result to avoid recomputation and never releases it, so executor memory that the shuffle needed is held by a result nobody reads any more (Memory Pressure, Swap and the OOM Killer).
What is actually happening
- Each transformation returns a new description that references its parent. Nothing is computed and nothing is read; the driver accumulates a graph (DAG (Directed Acyclic Graph)).
- An action — write, count, collect, show — asks for a result. Only then does the driver resolve the plan against the catalog, optimise it, cut it into stages and schedule tasks (Stages and Tasks).
- Deferring is what makes whole-query optimisation possible. Because the engine sees the filter and the projection and the join together, it can push the filter into the scan and drop the columns nobody uses — none of which is possible if each step executes as it is written (Query Optimizers).
- The plan is a description, not a cache. Calling a second action re-executes the plan from its sources, because nothing between them retained the intermediate result. Persisting is the explicit instruction to retain it.
- Errors that depend on data — a bad cast, a missing file, a duplicate key — can only be discovered when data is read, which is at the action. Errors that depend on schema are found earlier, at plan resolution, which is why some mistakes fail fast and others do not (Schema Evolution).
Nothing has happened yet
The first thing to internalise is that the object you are holding is not data. It is a description of how to produce data, and until an action is called, the cluster has done nothing at all: no file has been opened, no row has been read, no task has been scheduled.
This is why a transformation chain returns instantly regardless of how much data it names, and why a mistake in the middle of it says nothing until later. It is also why the optimiser can do anything useful: it gets the whole chain at once, so it can push a filter that you wrote last into the scan that you wrote first (Predicate Pushdown).
The consequence that costs money is the second one below. Two actions over the same chain are two full executions, from the source. The engine did not keep the intermediate result because nobody asked it to, and the log line that printed a count did as much work as the write that followed it.
- 1Transformation call
Appends a node to the plan and returns immediately.
guarantees No data access, no cost, no error that depends on the data.
fails by Silently accepting a mistake that will only surface much later, at the action.
- 2Action call
Triggers resolution, optimisation, planning and scheduling.
guarantees A result computed from the plan as it stands at this moment.
fails by Reporting every error here, whatever caused it, which is why traces point at the wrong line.
- 3Resolution against the catalog
Binds names to tables and columns and checks types.
guarantees Schema errors are caught before any task runs — the one place a lazy engine fails fast.
fails by Not catching anything that depends on values rather than on types.
- 4Execution
Runs stages and tasks over the data.
guarantees Nothing about how many times a step runs: retries and second actions both re-execute.
fails by Producing inconsistent output when the plan reads a mutable source at two different moments (Determinism: Same Input, Same Output?).
- 5Persist / materialise
Retains an intermediate in memory, or writes it to storage.
guarantees Memory retention is best effort and can be evicted; a storage write is durable and is a restart point.
fails by Taking memory the shuffle needed, or being forgotten and never released (Memory Pressure, Swap and the OOM Killer).
Only two rows here guarantee anything durable, and both of them are things you have to ask for. Everything else is a description or an attempt.
1orders = (2 spark.read.parquet("s3://lake/orders") # nothing read yet3 .filter("dt = '2026-08-25'") # nothing filtered yet4 .filter("status <> 'cancelled'") # still just a plan5 .join(customers, "customer_id") # still just a plan6)7 8print(orders.count()) # ACTION 1: full scan, filter, join9orders.show(5) # ACTION 2: does it all again10orders.write.parquet(out) # ACTION 3: and again11 12# Same chain, three executions. The two debugging lines cost as much13# as the write. Persisting turns three executions into one plus two reads:14 15orders.persist() # a memory commitment, not a hint16print(orders.count()) # ACTION 1: computes and retains17orders.write.parquet(out) # ACTION 2: reads the retained result18orders.unpersist() # release it, or it holds memory the shuffle needsThe trap is not the persist call, it is the print. A debugging action left in scheduled code doubles the job forever, and nothing in the run report says "this ran twice" — it simply appears as two jobs where you thought there was one.
Debugging a plan you cannot see
count() on a filtered chain can execute a very different plan from the write() that failed, because the optimiser prunes columns the count does not need. It usually finds the mistake anyway, and when it does not, the physical plan is the authority.The practical cost of laziness is that the error message is systematically unhelpful: it names the action, because that is where execution was requested, and the mistake is somewhere in the graph behind it. The fix is a method rather than a tool.
Work in halves. Apply a cheap action to the first part of the chain; if it succeeds, the mistake is downstream. Repeat. This finds a failing transformation in a handful of steps, and it is far faster than reading a long trace that describes the scheduler rather than the query.
The second technique is to read the plan itself. The physical plan shows the operations in execution order, with the pushdowns applied and the join strategies chosen — which is a different and more honest document than the code, and it is the artefact worth attaching to any question about why a job is slow (Reading EXPLAIN ANALYZE).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A cast error on a column set twenty lines earlier. | The stack trace names write(). | Nothing executed until the write; the failing expression is anywhere in the plan. | Bisect the chain with a cheap action, or read the physical plan to find where the expression sits. |
A debug count() left in a scheduled job. | Runtime and cost roughly double; no error, no warning. | Two actions over one chain means two executions from the source. | Remove debugging actions from scheduled code, or persist deliberately if the reuse is genuine (Compute Waste). |
| A chain reads a table that another job is writing. | Output that is internally inconsistent — a join whose two sides reflect different moments. | Two executions of the plan read the source at two different times. | Pin the source to a snapshot or a table-format version, so both executions see the same state (Open Table Formats). |
| A persisted result under memory pressure. | The "cached" chain recomputes anyway, and everything else spills more than before. | Cached blocks were evicted, and the memory they held was taken from the shuffle in the meantime. | Persist selectively, release explicitly, and materialise to storage for anything large or long-lived (Checkpointing). |
How to build it
Most important first.
- Know which calls are actions. Everything else is free; every action is a job with a full execution behind it.
- Materialise or persist a result that will be consumed more than once, and release it when the chain is done. Persisting is a memory commitment, not a hint (Checkpointing).
- Prefer writing an intermediate to storage over caching it in memory for anything long-lived. It survives executor loss, it is inspectable, and it is a restart point (Reprocessing vs Retrying).
- Keep debugging actions out of scheduled code. A
count()that was helpful during development is a duplicate execution in production (Compute Waste). - Make the plan deterministic — no wall clock, no random seed, no mutable source read mid-run — so that two executions of the same plan cannot disagree (Idempotent Data Pipelines).
- When an error is confusing, run the chain in pieces with a cheap action at each step to find where it actually breaks. The trace tells you where execution was requested, not where the mistake is (Debugging a Data Incident).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The engine guarantees the result of the plan, not the order or the number of times its steps run. A stage may execute more than once due to retry or due to a second action (Determinism: Same Input, Same Output?).
- It guarantees whole-plan optimisation is possible; it does not guarantee any particular rewrite happens, and it cannot rewrite through opaque expressions (Query Optimizers).
- A cached result is a best-effort retention. Blocks can be evicted under memory pressure, and their loss triggers recomputation rather than an error — so caching improves the expected case and promises nothing.
- Nothing guarantees that two executions of a plan produce the same rows when a source is mutable. That is the caller's responsibility and it is the one people are surprised by (Source of Truth).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Assert that a job performs the number of actions you expect. Counting jobs in the run's metadata is a one-line check that catches a stray debug action added months ago (Pipeline Metrics).
- Reconcile the output against the source for the period. A plan that read a mutable source twice produces an output that reconciles against neither state cleanly, which is how this failure becomes visible (Reconciliation).
- Both miss a plan that is deterministic, executed once, and semantically wrong. Laziness is a mechanism, and mechanisms do not check meaning.
- Laziness does not affect freshness directly; it affects how much work is done to achieve it. A plan executed twice takes roughly twice as long to land the same output.
- A plan that reads a source at two different moments can produce an output mixing two states of that source, which is a freshness *inconsistency* rather than a delay — and it is invisible in every freshness metric (Freshness Checks).
- Materialising an intermediate makes downstream steps start from a fixed snapshot, which trades storage for a well-defined moment in time (Snapshot Tables).
- Adding a step to a chain costs nothing until an action runs, so the cost of a change is not visible at the point of the change. Reviewers should read the plan, not the diff, for anything performance-sensitive (Reading EXPLAIN ANALYZE).
- A schema change upstream is caught at plan resolution rather than at execution, which is the one place a lazy engine fails early and helpfully.
- Adding a second consumer of an existing chain silently doubles its execution. That is an architectural change disguised as a one-line addition (Model Layering).
- Nothing persists, so there is nothing to repair. Recovery from a lazy-evaluation mistake is a code change and a re-run.
- When two executions of a plan disagreed because of a mutable source, the recovery is to pin the source — a snapshot, a bounded range, a table format's version — and re-run (Open Table Formats).
- Materialising the expensive intermediate converts a re-run of everything into a re-run of the tail, which is the practical form of recovery for long chains (Checkpointing).
What can go wrong
- An error message that names the action rather than the transformation that caused it.
- Silent duplicate execution of an entire chain, visible only as cost and duration.
- A cached result evicted under memory pressure, so the "cached" chain recomputes anyway — slower than if it had never been cached, because the caching itself cost memory.
- A plan reading a mutable source at two different times, producing an internally inconsistent output with no error anywhere (Atomic Publish).
- The mitigation failing: caching everything, which starves the shuffle of memory and makes the job slower overall (Memory Pressure, Swap and the OOM Killer).
- "The variable holds my data." It holds a plan. Nothing has been read, and passing it around passes a recipe rather than a result.
- "The error is where the stack trace points." The trace points at where execution was requested. The mistake is somewhere in the plan behind it.
- "Caching makes things faster." Caching makes *reuse* faster and makes everything else slower by taking memory. Cached once and read once is pure overhead.
- "Two runs of the same code give the same answer." Only if the plan is deterministic and the sources are stable. Neither is automatic (Determinism: Same Input, Same Output?).
Operating it
- Job count per run. One action is one job, so the number of jobs is the number of times something was demanded (Pipeline Metrics).
- Repeated identical stages across jobs in the same application — the signature of a chain being executed more than once.
- Cache hit and eviction statistics, which say whether a persist is doing what it was added for (A 95% Hit Rate Tells You Almost Nothing).
- Total input bytes read per run against the source size. Reading the source twice is unmistakable here and invisible everywhere else (Scan Cost).
- At 10x, duplicate execution costs 10x more, so a habit that was tolerable becomes a line item.
- At 100x, caching in memory stops being viable for most intermediates and materialising to storage becomes the default for anything reused (The Lakehouse).
- Plan size itself grows with the number of transformations, and very large plans make the driver's optimisation phase noticeable — a rare failure and a real one at extreme chain lengths.
- Duplicate execution is the dominant cost of misunderstanding laziness, and it is proportional to the whole chain rather than to the extra action (Compute Waste).
- Caching costs executor memory that the shuffle and the aggregation would otherwise use, so an over-eager persist makes everything else spill (The Shuffle).
- Materialising to storage costs a write and buys a restart point plus a fixed snapshot. For a long nightly chain that is usually the better trade.
- Laziness buys whole-query optimisation and gives up straightforward debugging. That is a good trade and it is the reason the error messages are the way they are.
- Persisting buys reuse and costs memory that other parts of the job need; materialising buys durability and costs a write.
- Breaking a chain into materialised steps makes the pipeline debuggable and inspectable, and gives up cross-step optimisation the engine could otherwise have done (Model Layering).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ENGINE-SPECIFICDeferred execution with an explicit action is Spark's and Flink's DataStream model; a SQL engine like Trino or a warehouse defers implicitly within a statement and executes on submission, so the debugging consequences differ even though the optimisation benefit is the same.
- GENERALEvery declarative query system separates description from execution — that separation is what an optimiser needs to exist at all. What varies is whether the boundary is visible to the programmer as an action call or hidden behind statement submission.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the review discipline that would catch a debugging action left in scheduled code, and the practice of treating a plan change as a change worth reviewing.