What Actually Drives Data Platform Cost
Storage, scans, shuffle, compute hours, network, retention, file count and repeated work — put in the order they actually move the number.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A platform's spend grew faster than its data did. Which drivers moved, and how would you find out which one rather than guessing?
Two audiences who want different things. Whoever pays for the platform wants a defensible answer to "why did this go up", and they will accept a driver breakdown but not a shrug. The engineers who will be asked to fix it want an ordering — which three changes are worth a sprint and which twenty are rounding — because a list of eighteen optimisations with no ranking produces the same result as no list at all.
The unit of cost in an analytical platform is not a job and not a table. It is one unit of chargeable work attributed to one dataset: one byte scanned, one byte shuffled, one byte held for one day, one metadata request, one worker-second held open. Every driver below is one of those units multiplied by how many times something causes it to happen, and the multiplier is almost always the part that moved.
Watch the monthly total. When it rises, ask people to be careful, delete a few unused dashboards and shorten one retention window. This is a completely reasonable first response and it usually produces a real reduction, because the first pass over an unexamined platform always finds something. It stops working the second time, and the reason is that it never established which driver was dominant — it just removed whatever was easiest to see.
The total goes back up within two months and nobody can say what caused either the fall or the rise, because no measurement was attached to either change. The platform now has a cost *story* rather than a cost model.
- The total goes back up within two months and nobody can say what caused either the fall or the rise, because no measurement was attached to either change. The platform now has a cost *story* rather than a cost model.
- The easy wins were all in storage, which is usually the smallest of the large drivers, while the actual growth was in a nightly job that rebuilds three years of history to add one day of it (Full Refresh vs Incremental).
- A dashboard is deleted, spend does not move, and the team concludes dashboards are not the problem. In fact that dashboard read a small aggregate table; the expensive readers were four scheduled extracts nobody thought of as queries (Scan Cost).
- Retention is shortened on the raw zone — the one layer that is both cheap to keep and the only thing that makes reprocessing possible — and six weeks later a transformation bug cannot be corrected because the inputs are gone (Keeping Raw History: The Recovery Position and the Liability).
- A team switches to a smaller cluster. Jobs now take longer, the shuffle spills to disk, and the same work costs more hours than before. Compute was reduced along the wrong axis, because nobody separated "how much parallelism" from "how much work" (The Shuffle).
- Someone partitions a large table by a high-cardinality key to make queries cheaper. Bytes scanned fall, file count explodes, and every query — including the ones that read almost nothing — now pays for hundreds of thousands of metadata requests (Partition Cardinality).
What is actually happening
- Almost every charge in a data platform reduces to one of four physical events: bytes read, bytes moved between machines, bytes held over time, and machine-time held open. Everything a vendor bills you for is a way of counting one of those, which is why the drivers transfer across products even though the pricing models do not.
- Those events are multiplied by two counters that are invisible in any single run: how many consumers cause it and how many times it is repeated. A transformation that reads a year of history is not expensive; a transformation that reads a year of history every night for three years is, and the code looks identical on any given day (Compute Waste).
- Storage is the driver everyone thinks of first and it is usually the *slowest-moving* of the big ones, because it grows with the data. Scans and repeated compute grow with the number of consumers and the schedule, which are organisational variables — and organisations grow faster than datasets do.
- File count is a driver in its own right and it is not proportional to bytes at all. Object stores charge per request, and a query over a million small objects spends its life in listing and open calls while reading almost no data (File Size and the Small-Files Problem).
- Network cost is mostly a topology question rather than a volume one. Moving bytes within a zone is generally free or near-free; moving them across a region boundary or out of the provider is a separate, much larger line, and the architecture decides how often that happens rather than the query (Egress: Moving Data Costs Money, Not Just Storing It).
- The pricing model decides which of these you feel. On a scan-priced warehouse an idle cluster costs nothing and a careless
SELECT *costs a great deal; on a time-priced cluster the same query is free at the margin and the idle hours are the bill. The same code has opposite cost behaviour on the two models, which is why cost advice copied from another company is so often wrong (Comparing Analytical Warehouses).
The drivers, in the order they usually move
The list of things a data platform can be charged for is long and mostly uninteresting. The useful artefact is an *ordering*: given a platform you have never seen, which drivers are probably dominant, and which ones are people's intuition rather than the bill.
The ordering below is the one that holds most often on a platform of moderate size with a mixture of scheduled pipelines and human consumers. It is not a prediction and it is deliberately unitless. What transfers is the relationship between the bars and the note attached to each one, because the note names the design decision that moves it.
Two things about this ordering are worth arguing with, and both arguments are productive. The first is that repeated work sits at the top when nobody ever describes their platform that way — every team believes its pipelines are incremental until someone measures rows read against rows written. The second is that storage, the driver most people name first, sits near the bottom: it grows with the data, and the data is the one thing on this list that grows slowly and for a good reason.
A nightly full rebuild does work proportional to all of history, every night, while the amount of new data per night is flat. Moved by converting the heaviest models to incremental processing with an explicit watermark.
Decided by column selection and by whether the predicate matches the partition key. Long-tailed: a small number of readers, often scheduled ones rather than humans, dominate the whole distribution.
Dominant on time-counted platforms and near zero on scan-counted ones. Moved by right-sizing, by autoscaling that actually scales down, and by not holding a cluster for a job that runs for a few minutes an hour.
Grows with join fan-out and with skew. One hot key can make a single worker's share dominate the whole stage, so the driver is cardinality distribution rather than total size.
Independent of bytes entirely. Rises with partition cardinality and with append frequency, and is the driver that a bytes-based cost report cannot see at all.
Accumulates because nothing ever prompts a review. Time-travel versions, failed-run output and abandoned experiment tables are all storage nobody has decided to keep.
A topology property, not a query property. Near zero if compute sits beside storage, and a distinct line the moment a consumer, a replica or a vendor integration lives somewhere else.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights on a modelled platform, published to establish an ordering and nothing else. The teaching is in which levers are attached to which bar: the top two are moved by decisions an engineer can make this week, and the bottom two are architecture.
What each driver is actually counted in
The reason cost conversations go badly is that people compare drivers that are not measured in the same thing. "We should store less" and "we should scan less" sound like the same kind of suggestion, and they are not: one changes a quantity that persists and the other changes a quantity that repeats.
The table below separates the counted unit from the thing that moves it. The middle column is what a platform bills; the right column is what an engineer can actually change. Almost every failed cost programme is one where somebody optimised the middle column directly — by capping, quota-ing or forbidding — instead of changing the thing in the right column that produces it.
Notice how many of the right-hand entries are decisions made long before anyone looked at cost. The partition key, the refresh schedule, the layer structure and the region topology are all set early, cheaply, by someone reasoning about correctness or latency. That is why cost engineering in this domain is mostly archaeology, and why the highest-leverage moment to think about it is the design review, not the invoice.
- The last row has no line item of its own, and that is the point: repetition multiplies every other row, so it appears in the bill as a slightly larger version of everything.
- A driver you cannot attribute to a dataset is a driver you cannot act on. Attribution is not reporting overhead, it is the mechanism by which any of these becomes someone's problem (Cost Attribution).
- Two drivers move in opposite directions when you fix one: shrinking files to prune better raises request count, and compacting to lower request count temporarily raises compute (File Compaction).
| Driver | Counted as | Moved by | Where it hides |
|---|---|---|---|
| Scans | Bytes read from storage to satisfy a query | Column projection, predicate that matches the partition key, pre-aggregated serving tables | Scheduled extracts and BI tools that issue SELECT * against a view nobody reads directly |
| Shuffle | Bytes written to and read from the exchange between stages | Join order, broadcast of the small side, key distribution, pre-bucketing | One skewed key, which shows up as a long job rather than as a large number |
| Compute | Worker-seconds held, running or idle | Right-sizing, scale-to-zero, batching small jobs, removing jobs with no consumer | A cluster kept warm "for interactive use" that is interactive for one hour a day |
| Storage | Bytes held, per unit of time | Retention policy per layer, tier transitions, deleting failed-run output and abandoned tables | Table-format snapshots and time-travel versions, which are real bytes with no listing that shows them |
| Requests | Metadata and object operations, counted per call | File size targets, compaction, partition cardinality | A query that reads almost no data and still takes minutes, which reads as a performance problem |
| Network | Bytes crossing a zone, region or provider boundary | Co-locating compute with storage, replicating once instead of reading across, batching vendor exports | A single cross-region dashboard, or a reverse-ETL job that ships the same table out every hour |
| Repetition | The multiplier on every row above | Incremental processing, longer intervals for low-value refreshes, retiring unused models | Nowhere — it is not a line item on any bill, which is precisely why it is usually the largest one |
How a platform gets expensive without anyone deciding
No engineer has ever chosen to spend money on recomputing 2019. Expensive platforms are not the result of bad decisions; they are the result of good decisions accumulating without anyone re-examining them, because every individual step was locally reasonable and nothing in the system ever pushes back.
Each row below is a real sequence. The trigger is small and defensible, the symptom appears months later somewhere else entirely, and the cause is a decision that was correct at the time it was made. The response column is deliberately specific: the general advice "be more careful" has never fixed any of them.
The common structure is worth naming. In every case something became repeated or wide without a review point — a job moved to a shorter schedule, a table grew a column, a dashboard was cloned, a partition key was chosen for a query pattern that later changed. Cost engineering is largely the practice of installing review points on those four events.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A model is switched from weekly to hourly because one stakeholder wanted fresher numbers. | Compute spend rises steadily over the following month with no change in data volume. | The job is a full rebuild. Its cost is proportional to history, and the schedule multiplied that by roughly one hundred and sixty-eight without changing a line of its logic. | Make the model incremental before changing its schedule, and state the freshness requirement as a decision the requester co-signs (Cost vs Freshness). |
| A wide raw table is exposed directly to the BI tool because building a serving model would have taken a week. | Scan-counted spend grows with headcount rather than with data. | Every dashboard reads every column of the raw table, and BI tools generate SELECT * against views by default. There is nothing between the consumers and the widest table in the platform (Data Marts). | Put a narrow, pre-aggregated serving table between the raw layer and the tool, and measure bytes scanned per dashboard afterwards to confirm the readers actually moved (Model Layering). |
| A streaming writer lands events as they arrive so the raw zone is always current. | Every query over that dataset becomes slow, including ones that filter to a single hour, and request-counted charges rise sharply while bytes stay flat. | A file per micro-batch produces an enormous object count. The cost has moved out of bytes read and into listing and opening objects (File Size and the Small-Files Problem). | Schedule compaction on the raw prefix and set a file-size target on the writer. Accept that compaction itself is compute you are now paying for, deliberately (File Compaction). |
An analyst partitions a fact table by user_id after reading that partitioning makes queries cheaper. | Queries that filter by date get slower and more expensive; queries that filter by user get faster. | Partitioning only prunes for predicates that match the partition key, and a high-cardinality key produces one tiny partition per value — moving the cost into metadata rather than removing it (Partition Cardinality). | Partition on the predicate almost every query carries, usually date, and use clustering or sort order for the secondary key (Clustering and Sort Order). |
| A team builds an experimental model, gets an answer, and moves on. | Three years later the platform holds several hundred tables, and a materially large share of the nightly run produces data nobody reads. | Nothing retires a dataset. Creation is self-service and deletion requires proving a negative, so the population only grows (Data Ownership). | Record days-since-last-read per dataset, keep the history longer than your longest-period consumer, and make retirement the default for anything unread past that horizon (The Data Catalog). |
| A vendor integration is added that pulls a table into a third-party tool every hour. | Network charges appear as a new line, and the source table's scan cost roughly doubles. | A reverse-ETL consumer is a full-table reader on a schedule, crossing a provider boundary each time. It is invisible in dashboards because it is not a dashboard (Egress: Moving Data Costs Money, Not Just Storing It). | Give the integration an incremental contract — a watermark column and a narrow projection — and count it as a first-class consumer in attribution (Incremental Extraction). |
How to build it
Most important first.
- Build the driver breakdown before you optimise anything. Until you can say roughly what share of spend is scans, compute hours, retained bytes and requests, every change is a guess with a plausible story attached (Cost Attribution).
- Attack repeated work first. It is the only driver whose multiplier you can reduce without reducing what anyone can ask, and in most platforms it is the largest single line (Incremental Processing).
- Then attack the scan shape: column selection and partition pruning on the heaviest reader queries. These are the cheapest fixes in the domain — they change a
WHEREclause and aSELECTlist, not an architecture (Scan Cost, Partition Pruning). - Fix layout before buying compute. A job that is slow because one partition holds most of the rows does not get cheaper with more workers; it gets more expensive, because more workers wait for the same task (Data Skew, Straggler Tasks).
- Put retention on a schedule with an owner and a stated reason per layer, rather than as a one-off cleanup. Retention is the driver that only ever grows if nobody is responsible for it (Storage Lifecycle, Data Retention).
- Give every dataset a cost line, not every person. A dataset is a durable thing with an owner and a dependency graph; a person moves teams, and a query attributed to an analyst tells you nothing about which model made it expensive (Cost Attribution).
- Re-derive the ordering after each change. The largest driver moves when you fix it, and the second-largest is frequently a different *kind* of problem — this is bottleneck migration, and it applies to spend exactly as it applies to latency (The Bottleneck Moves After Every Fix).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A platform guarantees you a unit of charge, not a price and not a bound. It will tell you what it counts — bytes scanned, worker-seconds, requests, stored bytes — and counting is the only thing that is stable enough to design against.
- Nothing guarantees that a query costs what it cost last week. The same SQL over a table that has grown, or been repartitioned, or lost its clustering to a hundred small appends, reads a different amount (Clustering and Sort Order).
- No system guarantees that spend and value are related. A dataset with no consumers costs exactly as much to store, refresh and back up as one that the company runs on (Data Discovery).
- Nothing guarantees the drivers stay in the same order. A platform that is scan-dominated today becomes compute-dominated the moment someone lands a streaming pipeline, and the previous quarter's optimisation playbook stops applying (Cost vs Freshness).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that catches cost regressions is a per-dataset spend trend against its own history: for each serving dataset, the chargeable units attributed to producing it and to reading it, compared week over week. A step change on a dataset whose row count did not step is a layout or query regression, and it is visible days before anyone looks at a total.
- It misses gradual drift completely — the case where every week is a little more than the last, which is exactly what an accumulating full rebuild looks like. Pair it with a check on the ratio of rows written to rows read by the producing job, which is flat for an incremental pipeline and grows without bound for a rebuild.
- It also misses everything about value. A dataset can be cheap, fresh, correct and completely unused, and no cost check will ever object. That question is answered by read counts and by lineage, not by a cost metric (Impact Analysis).
- Freshness is a cost driver disguised as a requirement. Every reduction in the interval between runs multiplies the fixed part of each run — planning, cluster start, reading the previous state, publishing — by the number of runs, and that fixed part does not shrink with the interval (Cost vs Freshness).
- Cost work itself changes freshness in one direction that matters: compaction, clustering and lifecycle transitions are jobs that must run somewhere, and they compete with the pipeline for the same window and the same capacity (File Compaction).
- The honest framing for a consumer conversation is not "this is expensive" but "this is the frequency you asked for; here is what a longer interval would return to you and what it would cost you in staleness" (The Freshness SLO).
- Schema changes move cost. Adding a wide column to a table that consumers read with
SELECT *raises every one of those queries at once, and nothing about the change looks like a cost decision (Projection Pushdown). - Changing a partition key is a cost decision with a migration attached: the old layout must be rewritten and every reader whose predicate matched the old key silently stops pruning (The Partitioning Decision).
- Meaning changes cost the most, indirectly. When a metric's definition changes, the old model is usually kept "for comparison" and never retired, so the platform now maintains two of everything, forever (Semantic Changes, The Metrics Layer).
- Cost mistakes are among the most recoverable failures in this domain — with one exception. Over-scanning, over-provisioning and over-refreshing are all reversible by changing a query, a size or a schedule, and the spend stops the moment you do.
- Deleting data to save money is the irreversible one. A retention change applied to raw history cannot be undone, and it removes the ability to reprocess, which is the property that makes every *other* mistake recoverable (Keeping Raw History: The Recovery Position and the Liability, Reprocessing vs Retrying).
- Before any deletion, take an inventory of what reads the layer — including quarterly and annual jobs, which by definition have not run recently and therefore look unused to any thirty-day access report (Data Lineage).
- When a cost fix breaks something, the fix must be revertible independently of the pipeline it lives in. A "cost optimisation" merged into the same change as a logic change is a rollback you cannot perform (Rolling Back Data).
What can go wrong
- Optimising the visible driver rather than the dominant one, because storage has a number on the front page and repeated compute does not.
- A cost cut that reduces spend and correctness together — sampling an input, dropping a dimension, shortening a window — where only the spend was measured afterwards.
- Deleting or archiving data that a low-frequency consumer depends on, discovered a quarter later when that consumer runs (Storage Lifecycle).
- The mitigation failing: a cost dashboard that attributes everything to the service account the orchestrator runs as, so every job in the platform appears as one enormous, unactionable line (Cost Attribution).
- A freeze on new datasets as a cost control, which stops the cheap new work and leaves the expensive existing work untouched, while pushing teams to build shadow pipelines outside the platform (Data Platform Anti-Patterns).
- Cost pressure applied to the team that runs the platform rather than to the teams that generate the load, which produces heroic infrastructure tuning and no change in behaviour (Who Owns Data Quality).
- "Storage is the biggest cost, because it is the biggest number of bytes." Bytes held and bytes read are charged by different mechanisms and grow for different reasons. A modest table read by two hundred dashboards outweighs an enormous archive nobody touches.
- "We reduced spend, so we improved the platform." Only if what was removed had no consumer. A reduction that came from a shortened retention window or a sampled input is a transfer of cost into a future incident.
- "The expensive query is the slow one." Runtime and chargeable work are related but not the same. A query that reads an enormous amount in parallel can be fast and expensive; one that waits on a single straggler can be slow and cheap (Straggler Tasks).
- "This job is cheap, it runs in a couple of minutes." Multiply by the schedule before believing it. A short job on a five-minute schedule does far more work in a month than a long one that runs weekly (Cost vs Freshness).
- "Compute is the expensive part, so we should cache more." Caching moves work; it does not remove it, and a cache over a dataset that changes every run is pure overhead (When Not to Cache).
Operating it
- Chargeable units per dataset per day, split by driver — produced-by versus read-by. This one table answers most cost questions and almost nobody builds it before the first bad month (Cost per Service and the Attribution Problem).
- Bytes scanned per query, ranked, with the query text and the caller. The distribution is nearly always long-tailed: a handful of readers dominate, and they are frequently scheduled extracts rather than humans (Scan Cost).
- Rows read divided by rows written per pipeline run. Flat means incremental; growing means the job is recomputing history (Incremental Processing).
- File count and mean file size per dataset over time. A rising count with flat bytes is the small-file pathology arriving (File Size and the Small-Files Problem).
- Days since last read per dataset, from query history. It is the only evidence that lets you retire anything, and it must be kept for longer than your longest-period consumer (The Data Catalog).
- At 10x data volume, storage and scan both scale with the data, and partition pruning stops being an optimisation and becomes the thing that decides whether a query is possible at all (Partition Pruning).
- At 10x consumers, the scan driver scales and nothing else does — which is why platforms that grew by adding teams look completely different from platforms that grew by adding rows, and need opposite fixes.
- At 100x, full rebuilds are no longer schedulable at any price and every pipeline becomes incremental, which converts a cost problem into a correctness problem: watermarks, late data and idempotency (The High-Water Mark, Late-Arriving Data).
- Cardinality scales differently from volume. Ten times the distinct keys can leave bytes unchanged while multiplying partitions, files and metadata requests — the driver that is invisible on a bytes-based cost report (Partition Cardinality).
- The four physical drivers: bytes read by queries and jobs, bytes moved between machines by shuffles and across network boundaries, bytes held over time, and machine-time held open whether or not it is doing work.
- The two multipliers that decide which driver dominates: how many consumers cause the work, and how many times the same work is repeated. The multipliers move faster than the data does.
- File count as a driver independent of volume, because request-priced object stores charge for opening an object regardless of how little of it you read.
- The pricing model itself as a driver of which behaviour is expensive: scan-counted platforms punish wide reads and forgive idle capacity; time-counted platforms do the reverse.
- Every cost control is a constraint on someone's ability to ask a question. Aggregate tables, restricted retention, scheduled rather than continuous refresh — each one buys spend back by removing an answer someone could previously get, and pretending otherwise is how cost programmes lose the room.
- Attribution costs engineering time and produces no direct saving. It is a measurement system, and it competes with the fixes it would inform; the argument for building it first is that without it the fixes are chosen by whoever is loudest.
- Optimising a pipeline makes it more complicated. An incremental job with a watermark, a late-data window and a merge is genuinely harder to reason about than a full rebuild, and the full rebuild is more likely to be correct (Full Refresh vs Incremental).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe four physical drivers — bytes read, bytes moved, bytes held, machine-time held — are properties of the hardware and hold on every platform. What differs is which of them a given vendor counts and bills, and therefore which one you are able to see.
- WAREHOUSE-SPECIFICScan-counted and time-counted platforms reward opposite behaviour: on the first, idle capacity is free and a wide read is the bill; on the second, a wide read is free at the margin and the idle hours are the bill. Any cost advice that does not say which model it assumes is unusable.
- SCALE-SPECIFICBelow a few dozen datasets and a handful of consumers, a driver breakdown is more work than the spend it manages and the right answer is to look at the largest job by hand. Attribution becomes worth building at the point where nobody can name every pipeline in the platform.
- CLOUD-SPECIFICNetwork is near-free within a zone, meaningful across zones, and a separate line item across regions or out of the provider. Which of those boundaries your architecture crosses per query is a topology decision, and providers draw those boundaries in different places.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the tagging, budget alerting and account structure that make a driver breakdown enforceable rather than a spreadsheet somebody maintains by hand.
- — Distributed Systems owns why moving bytes between machines is a distinct and expensive class of work rather than an implementation detail of a join.