Cost vs Freshness
The most direct trade in the platform: every increment of freshness is bought with compute that runs more often, longer, or continuously. The right freshness is set by the decision the data drives, never by what the stack can achieve.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A consumer asks for this dataset to be fresher. What exactly gets more expensive, and what decision would have to change for that to be worth it?
The person asking for hourly instead of daily, who has never been shown what that costs, and the finance owner who sees the platform bill rise with no corresponding change in any decision anyone makes. Neither of them is wrong; they have never been in the same conversation (Cost Attribution).
The unit is one dataset, one refresh interval, one consumer decision. Cost and freshness are only comparable at that granularity: a platform-wide "we should be more real-time" has no unit and cannot be priced, argued with, or refused (Data Products).
Make everything as fresh as the tooling allows. Storage is cheap, the warehouse scales, the scheduler accepts any cron expression, and nobody was ever criticised for data being too recent. For a small platform this is genuinely fine and the alternative is premature optimisation.
The daily model becomes hourly. It was written as a full refresh, so it now rebuilds the entire history twenty-four times a day instead of once, and the scanned volume rises by the number of runs while the *new* data in each run is a twenty-fourth of a day (Full Refresh vs Incremental).
- The daily model becomes hourly. It was written as a full refresh, so it now rebuilds the entire history twenty-four times a day instead of once, and the scanned volume rises by the number of runs while the *new* data in each run is a twenty-fourth of a day (Full Refresh vs Incremental).
- The hourly runs each carry a fixed overhead — planning, warm-up, metadata commits, a validation pass — that does not shrink with the interval. Past a point the platform is paying mostly for starting and stopping (Compute Waste).
- Each run writes its own small output files, so a table that was compact becomes thousands of fragments and every downstream query pays for opening them. The freshness change made unrelated queries slower and more expensive (File Size and the Small-Files Problem).
- The team moves the pipeline to streaming for minute-level freshness. Batch cost was a burst that ended; streaming cost is a consumer that runs at three in the morning on a public holiday whether or not a single event arrives (Stream Processing).
- A mart is built to make the hourly dashboard cheap. It answers that one question beautifully and every adjacent question — the same metric by a different dimension — now requires either a second mart or a full scan of the fact table (Data Marts).
- Six months later nobody can name a decision that changed because the data got fresher. The dashboard is still read once each morning, and it is now recomputed twenty-four times to be ready for a person who arrives at nine (Who Actually Consumes This Data).
What is actually happening
- Freshness is bought with repetition. Halving the interval doubles the number of runs, and each run pays its fixed overhead again regardless of how little new data it has to process. That overhead is the term that makes very frequent batch inefficient long before the data volume does (Compute Waste).
- Whether the repetition is affordable depends entirely on whether the run is incremental. An incremental run processes new data and its cost scales with the interval; a full refresh processes everything and its cost scales with the number of runs. The same freshness change is cheap in the first case and close to linear in the second (Incremental Processing).
- Batch and streaming differ in cost *shape*, not merely in magnitude. Batch is a burst: capacity is held for the duration of a run and released. Streaming is a continuously running consumer: capacity is held always, sized for peak rate rather than average, and paid during every quiet hour (Batch vs Streaming Ingestion).
- Streaming adds costs batch does not have at all — state that must be kept for windows and joins, checkpoints written repeatedly, and a restore path whose duration grows with that state (Streaming State).
- A mart moves cost from read time to write time. Precomputing an aggregate makes one query pattern nearly free and makes every other question expensive, because anything the mart does not carry has to go back to the fact table. The mart is a bet on which question will be asked (Data Marts).
- Small files are the hidden term. Frequent writes produce many small outputs, which raise per-query overhead for everyone reading the table and create a compaction job that is itself a recurring cost (File Compaction).
- Freshness composes badly along a chain: making a leaf mart hourly does nothing unless every model above it is also hourly, so the requested change propagates upstream and multiplies across every node in the path (Model Layering).
- None of this is a reason to be stale. It is a reason for the interval to be a decision with a named beneficiary, because the interval is the single control that moves cost most directly and it is usually set by habit (The Freshness SLO).
What actually moves the number
When someone asks for fresher data, six things get more expensive and one gets cheaper. Knowing which is which is the whole skill, because the conversation that follows is otherwise a negotiation between a preference and a bill with no mechanism in between.
The bars below are relative and unitless. They say which drivers dominate the cost of buying freshness and what moves each one; they are deliberately not a measurement, because the actual weights depend on your engine's billing shape, your data volume and how much of the day is quiet. What transfers between platforms is the list and the notes, never the heights.
Two of these deserve attention because they are routinely forgotten. Fixed overhead per run does not shrink when the interval does, so at short intervals a growing share of the spend is starting and stopping rather than processing. And files produced per unit time exports part of the cost to every consumer of the table, appearing as a slow degradation in unrelated queries that nobody connects to a schedule change (File Size and the Small-Files Problem).
Moves inversely with the interval: halve the interval, double the runs. This is the driver the requester is implicitly asking to change, and it multiplies every other per-run driver below it (Orchestration).
Planning, cluster or container acquisition, metadata commits, the validation pass. It is constant per run, so its share of the total grows as the interval shrinks — which is what makes very frequent batch inefficient before data volume does (Compute Waste).
Moves with whether the run is incremental and whether its predicate prunes partitions. Incremental and pruning: nearly flat as frequency rises. Full refresh: multiplies by the number of runs (Partition Pruning).
Applies to a streaming consumer rather than a batch run. Sized for peak rate and paid during every quiet hour, so it is moved by the shape of the day rather than by the volume of data (Stream Processing).
Moves with window size, join bounds and key cardinality — not with freshness directly, but freshness is the reason the state exists at all, and it is written repeatedly at every commit (Streaming State).
Moves with write frequency and with parallelism. Raises per-query overhead for every reader of the table and creates a compaction obligation that is itself a recurring cost (File Compaction).
The one driver that moves *down* when you precompute. A mart converts many consumer scans into one write, which is what makes this a trade rather than a straight bill (Data Marts).
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Read the notes rather than the bars. A platform where every model is incremental and every consumer reads a mart has a completely different profile from one where daily full refreshes were simply scheduled more often, and both are drawn from this same list.
A burst, a continuous consumer, and a precomputed answer
There are three shapes a serving architecture can take, and they do not differ merely in how much they cost — they differ in *when* cost is incurred, which is what makes comparing them by a single figure meaningless. A batch run holds capacity for its duration and releases it. A streaming consumer holds capacity always. A mart moves the cost from the moment of reading to the moment of writing.
The claim that one of these is generally cheaper is not available. At low, bursty volume with a daily decision behind it, batch is obviously right. At high sustained rate with a continuous decision behind it, a streaming consumer pays its overhead once instead of thousands of times and frequently wins. Between those poles the answer depends on the shape of the day, the size of the state and how incremental the batch model is, which is why the comparison has to be made per pipeline rather than adopted as a policy.
The column that settles most real arguments is the last one. Operational burden is a cost that never appears on an infrastructure line item and is paid by the same small number of people every week — and a streaming job is never finished in the way a batch job is finished at 06:00 (Pipeline Reliability).
| Shape | When cost is incurred | What moves it most | Freshness it can offer | Failure and operational burden |
|---|---|---|---|---|
| Scheduled batch, daily | One burst per day, released afterwards. | Bytes scanned per run, and whether the model is incremental. | A settled figure for a closed period, available at a stated hour. | Lowest. A failure has a whole interval to be retried inside, and the job is genuinely finished when it finishes (Retries in Pipelines). |
| Scheduled batch, frequent | Many small bursts, each paying full fixed overhead. | Number of runs times fixed overhead, plus small-file production. | Bounded by the interval plus run duration — good, never continuous. | Moderate. Overlapping runs, partial publishes and compaction become standing concerns (Atomic Publish). |
| Continuously running consumer | Always, including hours when nothing arrives. | Provisioned peak capacity and retained state, not average volume. | Continuous, bounded by the completeness rule rather than by a schedule (Watermarks). | Highest. State, checkpoints, restore duration, watermark stalls and a job that must be operated rather than scheduled (Checkpointing). |
| Mart over any of the above | At write time, once per refresh, regardless of reads. | Refresh frequency times the cost of building the aggregate. | Its parent's freshness plus its own refresh interval — strictly staler than what it reads. | Adds a second dataset to keep consistent, and drifts from its parent whenever a definition changes in one (The Metrics Layer). |
| Query the fact table directly | At read time, once per query, multiplied by consumers. | Bytes each consumer scans, and how well the layout prunes (Partitioning). | Exactly the freshness of the underlying table. | None beyond the table itself, and it scales badly with consumer count rather than with data volume (Scan Cost). |
A mart makes one question cheap and every other question expensive
Precomputation is the only move in this lesson that reduces a cost driver, and it does so by fixing a grain and a set of dimensions in advance. That is the bet: you are wagering that the questions people ask will match the shape you chose, and the saving is real exactly to the extent that the bet is right.
The failure is not that the mart breaks. It is that the mart answers, confidently, a question it was not built for. A daily-by-country revenue mart asked for revenue by product returns nothing useful; asked for revenue by country for a partial day it returns a number that is smaller than the truth; and joined to another mart at a different grain it returns a number that is larger. None of those raise an error (Grain: What Does One Row Represent?).
The discipline is to write down what the mart is for, at which grain, with which dimensions — and to state what it cannot answer, next to the table where people find it. A mart with an undocumented grain becomes the default table for everything within a quarter, and the cost saving is repaid several times over in wrong numbers (Dataset Documentation).
| Stage | One row is | Breaks if |
|---|---|---|
| Raw order events | One state change of one order — created, paid, refunded, cancelled. | Summed as if each row were an order. A single order with four state changes contributes four times (Event vs Snapshot Modeling). |
| fct_orders | One order, at its final state for the period, with amount and dimension keys. | Joined to a payments table where an order can have several payments — the join fans out and every sum over it doubles (Fact Tables). |
| Mart: revenue by day and country | One day, one country, one summed amount. | Asked for anything not in the grain. Revenue by product is unanswerable; revenue for a specific customer is unanswerable; and a partial current day returns a real number that is quietly short (Data Marts). |
| Mart: revenue by day and product | One day, one product, one summed amount. | Joined to the country mart on day to get country-by-product. The join is at day grain and multiplies the two dimensions against each other, producing a total far above the truth (Star Schema). |
| Dashboard tile | One number, with the BI tool's own filters applied on top of whichever mart it reads. | A filter is applied that the mart's grain cannot support, so the tool silently aggregates what it has and the tile is confidently wrong (Two Dashboards, Two Numbers). |
Each precomputation buys cheapness by discarding the dimensions it did not keep. The saving and the limitation are the same act, which is why "what can this not answer" belongs in the mart's documentation rather than in the tribal knowledge of whoever built it.
Setting the interval from the decision
The question that ends most freshness debates is not technical: what would somebody do differently? A report read at nine each morning needs to be correct at nine each morning. An alerting rule that pages someone needs continuity, because a page four hours late is not a page. A monthly board pack needs a settled figure and would be actively harmed by an intraday one that keeps moving.
Once the decision is named, the interval usually names itself, and the conversation stops being about technology preference. The options below are the shapes that answer, ordered roughly by what they cost to run and to operate rather than by how modern they sound.
One rule survives every variation of this argument: never set the interval to what the platform can achieve. Technical capability is not a requirement, and a platform whose freshness is set by its own capabilities will spend continuously to satisfy a preference nobody has articulated and nobody will defend when the budget is reviewed (Data Platform Anti-Patterns).
A consumer wants fresher data. Which shape does their decision actually require?
when The decision is taken once a day at a known hour — a close, a morning report, a scheduled downstream job.
cost Nothing useful, for that consumer. It costs the ability to answer an urgent intraday question, which is a real limitation that should be stated rather than discovered (Pipeline SLOs).
when The decision recurs during working hours but is not continuous — a team checking progress, a spend allocation revisited at midday.
cost Multiplies fixed per-run overhead by the number of runs, and requires the model to be incremental before it is worth doing (Incremental Processing).
when Continuity is nearly required but the tolerance is minutes rather than seconds, and the engine's per-run overhead is small.
cost Fixed overhead becomes a large share of the interval, small files accumulate quickly, and a compaction job becomes mandatory rather than optional (File Compaction).
when The value genuinely depends on continuity — operational alerting, a customer-facing figure, a fraud rule. The decision is made by a system, not by a person on a schedule.
cost Capacity held during every quiet hour, state and checkpoints to operate, a restore path that grows with state, and a job that is never finished (Stream Processing).
when Most consumers want a cheap answer to a common question and a few need the underlying detail at full freshness.
cost Two datasets with two freshness figures, and consumers who will read whichever they found first. Only workable if both are documented and the difference is stated (Data Marts).
Which of these is cheapest on your platform depends on the engine's billing shape — capacity held versus work performed, per-run acquisition cost, whether idle streaming capacity can be scaled down — and every one of those has changed across recent versions of the major platforms. Verify the current behaviour of your own engine and measure your own pipelines; nothing in this lesson is a substitute for that, and any published comparison of magnitudes is out of date by the time it is read.
How to build it
Most important first.
- Start from the decision, not the capability. Ask what changes if the data is an hour old rather than a day old; if the honest answer is "nothing until tomorrow morning", the interval is settled and no further analysis is needed (Who Actually Consumes This Data).
- Make the model incremental before making it more frequent. Increasing the frequency of a full refresh is the most expensive way to buy freshness and the most common (Incremental Processing).
- Match the schedule to when the data is *read*, not to a round number. A report consumed at 09:00 needs one run that completes before 09:00, not twenty-four runs that keep a table warm for a reader who is asleep (Orchestration).
- Reach for streaming when the value is in continuous freshness — an operational view, an alerting rule, a customer-facing number — not when the requirement is one fresh answer at a fixed time. A frequent batch is usually cheaper and always simpler for the second case (Batch vs Streaming Ingestion).
- Build a mart only for a question that is asked often enough to repay precomputing it, and write down which questions it does not answer. An undocumented mart becomes the table everyone queries for things it was never built for (The Metrics Layer).
- Compact, or set target file sizes, whenever you increase write frequency. Otherwise the freshness improvement is partly paid for by every consumer of the table, in a line item they cannot see (File Size and the Small-Files Problem).
- Tier the platform explicitly: a small set of datasets with genuinely tight freshness and real objectives, and a majority on a daily or few-times-daily cadence. Uniform freshness across a platform is the most reliable way to spend a lot on very little (Pipeline SLOs).
- Publish the cost driver alongside the freshness objective so the two are negotiated together. A consumer asked "hourly, or daily and a third of the compute?" answers differently from one asked "how fresh would you like it?" (Cost Attribution).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Nothing here is a guarantee about magnitude. The relationship is directional — more frequent runs cost more, continuous consumers cost during idle periods, precomputation trades read cost for write cost — and the size of each effect is a property of your data, engine and query mix (What Actually Drives Data Platform Cost).
- A shorter interval guarantees more runs. It does not guarantee proportionally more useful output, and where the source produces nothing it guarantees runs that process nothing and are billed anyway (Compute Waste).
- A streaming pipeline guarantees capacity is held continuously. It does not guarantee lower total cost than batch, and it does not guarantee higher — that depends on rate, on state size and on how much of the day is quiet (Stream Processing).
- A mart guarantees a cheap answer to the queries it was shaped for. It guarantees nothing about any other query and it adds a second dataset with its own freshness, its own failures and its own maintenance (Data Marts).
- What is explicitly not guaranteed: that fresher data is more useful. That is a claim about a decision, and only the consumer can make it (Trusting Data).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Track cost per run and rows processed per run together, per pipeline. A rising cost with flat rows is a full refresh dressed as an incremental one, and it is invisible in any metric that looks at either number alone (Pipeline Metrics).
- Track reads per dataset alongside its refresh frequency. A table refreshed hourly and queried twice a week is the clearest signal a platform produces, and almost nobody has the join that shows it (Data Discovery).
- Both miss the case where the frequency is justified and the *model* is wasteful — a correctly hourly pipeline that rebuilds a year of history each time looks like a legitimate cost until someone reads the SQL (Scan Cost).
- They also miss the consumer-side cost of small files, which shows up as a slow degradation in unrelated queries and is attributed to the warehouse rather than to the schedule change that caused it (File Size and the Small-Files Problem).
- End-to-end freshness for a consumer is the sum of every hop plus the interval of the slowest one, so buying freshness at the leaf while the root runs nightly buys nothing at all (The Data Loop).
- Below a certain interval, batch stops improving freshness meaningfully because fixed per-run overhead becomes a large share of the interval itself. That is the point at which the architecture question — continuous consumer instead of frequent burst — becomes the real one (Batch vs Streaming Ingestion).
- Streaming freshness is continuous but not free of shape: a completeness boundary still has to advance, and the allowance for out-of-order data is a direct subtraction from how fresh a windowed result can be (Watermarks).
- A mart is always at least as stale as its inputs plus its own refresh, so adding a serving layer to make queries cheap costs freshness in the same move (Data Marts).
- Changing the interval changes the meaning of the data for anyone reasoning about it. A metric that used to be a settled daily figure becomes a moving intraday one, and consumers who compare a morning reading with an afternoon one will find a discrepancy that is not an error (Semantic Changes).
- Moving from batch to streaming changes the completeness rule from "the period published" to "the watermark passed", which changes what a number means without changing a single column (The Freshness SLO).
- Adding a mart adds a dataset that will drift from its parent whenever a definition changes in one and not the other. The cost saving is real and it is paid for with a second thing to keep consistent (The Metrics Layer).
- Reducing frequency later is organisationally much harder than increasing it. Freshness is experienced as a service level, and taking it back reads as a degradation regardless of whether any decision depended on it (Pipeline SLOs).
- A frequent schedule multiplies the cost of every reprocess: a corrected day must be recomputed across however many runs covered it, unless the unit of recomputation is the period rather than the run (Reprocessing vs Retrying).
- Make the recompute unit a business period — a day, an hour — independent of how often the pipeline runs, so a repair costs one unit rather than one per run (Idempotent Data Pipelines).
- Streaming recovery has a cost shape batch does not: restoring large state before processing resumes, during which the platform is paying for capacity that is producing nothing (Checkpointing).
- A mart is rebuilt from its parent, so a repair is two passes rather than one, in dependency order. Budget for that when deciding whether the mart is worth it (The Transformation DAG).
What can go wrong
- A full-refresh model put on a frequent schedule, so cost scales with the number of runs while the useful output does not (Full Refresh vs Incremental).
- Freshness bought at a leaf while an upstream model still runs nightly, so the spend produces no change a consumer can perceive (Model Layering).
- A streaming job provisioned for peak and paid for during a night with no traffic, replacing a batch job that finished in minutes (Stream Processing).
- A proliferation of marts, each built to make one dashboard cheap, collectively costing more to maintain and refresh than the scans they replaced (Data Marts).
- Small files from frequent writes degrading every query against the table, attributed to the engine rather than to the schedule (File Compaction).
- The mitigation failing: a compaction job added to fix small files, scheduled frequently enough that it becomes its own significant recurring cost (Compute Waste).
- An interval set once during a migration and never revisited, running for years against a decision that stopped existing (Data Platform Anti-Patterns).
- "Storage is cheap, so freshness is cheap." Freshness is not bought with storage. It is bought with compute that runs more often and, in the streaming case, with capacity held during hours when nothing arrives (What Actually Drives Data Platform Cost).
- "Streaming is more expensive than batch." Sometimes, and at high sustained rates frequently the reverse, because a continuous consumer pays its overhead once instead of per run. The comparison depends on rate, state size and how much of the day is quiet, and it has to be made per pipeline (Batch vs Streaming Ingestion).
- "We should be real-time." This is a statement about a technology, not about a decision. The useful version names the decision that would change and the person who would make it differently (The Freshness SLO).
- "A mart is a cost optimisation." It is a cost *relocation*, from read time to write time, and it is only an optimisation if the question it precomputes is asked much more often than it is refreshed (Data Marts).
- "We made it hourly and the bill barely moved, so frequency is not a driver." Check whether the model is incremental. An incremental model absorbs a frequency increase gently and a full refresh does not, and the same change on the next pipeline may behave completely differently (Full Refresh vs Incremental).
Operating it
- Refresh interval, cost per run and rows processed per run, on one row per pipeline. It is the join that makes wasteful frequency visible and most platforms have the three numbers in three different systems (Cost Attribution).
- Reads per dataset per week against its refresh frequency. Sorted by the ratio, the top of that list is the cheapest cost saving available to any platform (Data Discovery).
- Scanned volume per run over time, which reveals a model that has silently become a full refresh because an incremental predicate stopped pruning (Partition Pruning).
- Average output file size per write, trended. It is the leading indicator of the small-file problem, well before query times move (File Size and the Small-Files Problem).
- For streaming pipelines, provisioned capacity against actual rate across the day. The gap during quiet hours is the cost of continuity, stated plainly (Pipeline Metrics).
- At 10x data volume the fixed per-run overhead stops mattering and bytes scanned dominates, which makes incrementality the decisive question and makes frequency comparatively cheap (Incremental Processing).
- At 10x pipelines, uniform frequency becomes the largest single source of waste in the platform, and a tiering policy saves more than any query optimisation (Cost Attribution).
- At 100x event rate, a continuously running consumer is often cheaper than very frequent batch, because the fixed overhead is paid once rather than per run and the capacity is genuinely used. The comparison inverts, which is why it must be made rather than assumed (Batch vs Streaming Ingestion).
- At 10x consumers per dataset, precomputation earns its keep: a mart's write cost is paid once and its read saving is multiplied by everyone. The same mart serving one dashboard rarely pays for itself (Data Marts).
- The drivers that move with freshness are: number of runs, fixed overhead per run, bytes scanned per run, continuously held capacity, retained streaming state, and files produced per unit time. Every freshness decision moves at least three of them (What Actually Drives Data Platform Cost).
- The driver that moves *against* freshness is bytes scanned by consumers, which a mart reduces — which is why the trade is genuinely a trade and not simply a bill (Scan Cost).
- The relative weight of each driver depends on the engine's billing shape — capacity held versus work performed — and that is exactly the kind of thing that changes, so the transferable skill is knowing which drivers exist rather than which dominates on your platform this year (Separating Storage from Compute).
- The cost of getting this wrong is rarely a single large item. It is a hundred pipelines each running more often than anyone needs, which is why it is discovered during a budget review rather than by an alert (Cost Attribution).
- Frequent batch buys freshness at fixed granularity and costs repeated per-run overhead, more output files, and a compaction obligation. It is simple to operate and its cost grows in a way that is easy to forecast.
- A continuously running consumer buys genuine continuity and costs capacity during idle periods, state, checkpoints, a longer restore path and a materially harder operational model — including the fact that a streaming job is never "finished" (Streaming State).
- A mart buys cheap reads for one question shape and costs write-time compute, a second dataset to keep consistent, additional staleness, and an ongoing temptation for consumers to use it for questions it cannot answer (Grain: What Does One Row Represent?).
- Setting freshness from the decision buys a defensible platform and costs a conversation with every consumer, repeated whenever their decision changes. That conversation is the actual work of this lesson and there is no tool that performs it (Who Actually Consumes This Data).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat freshness is bought with repeated or continuous compute follows from what freshness is, so the direction of the trade holds everywhere; what varies is which driver dominates, and on a platform billed for held capacity rather than for work performed the idle hours of a streaming consumer matter far more than they do elsewhere.
- ENGINE-SPECIFICEngines differ in how much fixed overhead a run carries — planning, cluster acquisition, metadata commits — and that overhead is what decides at which interval frequent batch stops being sensible; an engine with near-zero start cost tolerates minute-level batch comfortably, while one that acquires a cluster per run does not, and the same schedule is therefore reasonable on one platform and wasteful on another.
- SCALE-SPECIFICBelow a handful of pipelines this is not worth managing and the correct answer is to run everything daily and revisit later; the tiering argument only earns its complexity once there are enough pipelines that uniform frequency is itself the dominant waste, which is a question of pipeline count rather than of data volume.
- SIMPLIFIEDThe drivers are treated as independent so each can be reasoned about separately. In practice they interact — more frequent runs produce more small files, which raise scan cost, which makes compaction necessary, which is itself compute — so the total effect of a frequency change is usually larger than the sum of the drivers considered one at a time.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the capacity and scheduling machinery underneath this — autoscaling policy, spot and preemptible capacity, cluster acquisition time — which is what decides how much a run costs before any of it reaches a data model.
- — Distributed Systems owns why a continuously running consumer cannot simply be scaled to zero and back without consequence: the state it holds has to be restored before it can resume, so idle capacity is buying restore-free continuity rather than nothing.