File Size and the Small-Files Problem
The same bytes split into a million objects behave nothing like the same bytes in a hundred. On object storage the cost is per request, so this is a problem about file count, not data volume.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The table holds the same data it held last month and the same total size, but every query against it got slower. What changed?
The analyst whose query now spends most of its time before reading anything, and the platform engineer who is asked why a small table is slow. Both need the answer to be a number they can see: not "the data grew" — it did not — but "the table is now made of far more objects than it was".
One file — one object in a bucket, one entry in a listing, one footer to read, usually one unit of work for one task. The whole lesson is the observation that a file has a fixed cost that is independent of how many rows are inside it.
Let the writer decide. A Spark job writes one file per output task, a streaming sink commits one file per micro-batch, an ingestion connector writes one file per API page. Nobody chose a file size; a file size emerged from the parallelism and the commit interval, and for the first weeks it is completely fine.
A streaming job commits every few minutes, seven days a week. After a year the table has hundreds of thousands of objects and a total size that would fit comfortably on a laptop. Query planning takes longer than query execution (Streaming Ingestion).
- A streaming job commits every few minutes, seven days a week. After a year the table has hundreds of thousands of objects and a total size that would fit comfortably on a laptop. Query planning takes longer than query execution (Streaming Ingestion).
- A Spark job runs with high parallelism against a small input, so it writes one tiny file per task. The next job reads that output with one task per file, inherits the fragmentation, and passes it downstream — fragmentation is contagious (Partitions: the Unit of Parallelism).
- An ingestion connector writes one file per source page. A slow day and a busy day produce wildly different file counts for the same schema, so the table's query cost has no relationship to its row count (Batch Ingestion).
- Someone partitions a modest table by an additional column "to make it faster". Each partition now holds a fraction of a day and each is written as at least one file. Total bytes unchanged; object count multiplied (Partition Cardinality).
- The table is on object storage, so listing it is a paginated API call. A
LISTover a prefix with a very large number of objects becomes a sequence of round trips before a single byte of data has been fetched (Direct Uploads and Signed Authorization). - Row-group statistics stop helping, because a file small enough to hold one row group has statistics that describe the whole file — there is no finer granularity left to skip within (Parquet Internals).
What is actually happening
- Every file carries fixed costs that do not shrink with its contents. It occupies an entry in a listing. It has a footer that must be fetched before anything inside it can be located. It is scheduled as a unit of work with task setup and teardown. It contributes an entry to whatever metadata layer tracks the table.
- On object storage those costs are billed and rate-limited per request, not per byte. This is the crux of the whole lesson: the pathology is a function of *how many objects exist*, and a dataset can be trivially small in bytes and pathological in request count at the same time (Object Storage).
- Reading a columnar file is at minimum two round trips — fetch the footer to learn where the column chunks are, then fetch the ranges you want. Both are requests. When the file is small, those two requests retrieve a payload smaller than the metadata that described it (The Parquet Read Path).
- Distributed engines usually assign work per file, so file count sets task count. Thousands of tasks each doing a few milliseconds of real work spend their time in scheduling, serialisation and result collection rather than in reading (Stages and Tasks, Parallel Overhead).
- Sequential access is destroyed. A large file is a long contiguous read that storage, network and prefetchers are all optimised for; the same bytes as many objects is many short reads with a request round trip in front of each (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax).
- The opposite extreme has its own mechanism. A file large enough that only a handful exist limits parallelism to that handful — the engine cannot split work below file granularity unless the format lets it read row groups independently, and even then the file must be opened by someone.
One dataset, two shapes
Take a dataset — call it one gigabyte, because the exact size does not matter and the ratio does — and write it two ways. First as a million objects of about a kilobyte each: one per streaming micro-batch, or one per source API page. Then as a hundred objects of about ten megabytes each. Identical rows, identical schema, identical total size, identical query results.
The two are not the same table in any way that a query engine cares about. Reading the first requires a paginated listing to enumerate a million keys, a million footer fetches to locate column chunks, and a million scheduled tasks each doing a kilobyte of work. Reading the second requires a short listing, a hundred footer fetches, and a hundred tasks each doing real work.
Notice what did not change: the bytes. Any monitoring built on data volume shows a completely stable table. The cost that exploded is a cost nobody was charting, which is why this problem is almost always discovered by a person complaining rather than by an alert.
The shape below is the whole lesson. It is worth being able to draw it from memory, because most conversations about "the lake is slow" end the moment someone counts objects.
events/date=2026-08-25/ events/date=2026-08-25/ part-000000.parquet 1 KB part-00000.parquet ~10 MB part-000001.parquet 1 KB part-00001.parquet ~10 MB part-000002.parquet 1 KB ... ... (100 objects total) part-999999.parquet 1 KB (1,000,000 objects total) same rows same rows same total size same total size 1,000,000 list entries 100 list entries 1,000,000 footer reads 100 footer reads 1,000,000 scheduled tasks 100 scheduled tasks
The cost is per request, not per byte
The reason this is a pathology rather than an inefficiency is that object storage prices and rate-limits requests. A GET for a kilobyte and a GET for ten megabytes are, from the store's point of view, both one request. Splitting the same data into more objects multiplies the number of requests needed to read it while leaving the bytes untouched (Direct Uploads and Signed Authorization).
That inverts the usual intuition. In most performance work, cost tracks volume: read more, pay more. Here you can hold volume constant and move cost by orders of magnitude purely by changing how the bytes are divided — and you can reduce volume without reducing cost at all, because the request count did not move.
Rate limits make it worse than a linear cost. A planning burst that issues an enormous number of listing and footer requests can be throttled, at which point the symptom is an intermittently slow query rather than an obvious storage error, and the investigation goes looking at the engine.
The drivers below are ordered for a fragmented table specifically. On a healthy table this ordering is different — bytes scanned dominates, as in Physical Data Layout — and that difference is the point: fragmentation does not make an existing cost worse, it introduces a cost that was previously negligible.
Scales with object count and is paid before any data is read. On a healthy table this is near zero; on a fragmented one it is the whole bill.
A fixed per-file cost independent of file size, which is precisely why it becomes dominant as files shrink.
One task per file doing almost no work. The engine looks busy and the cluster is doing bookkeeping (Parallel Overhead).
Unchanged between the two layouts. The thing everybody measures is the thing that did not move.
Essentially identical either way, plus a small per-object metadata overhead. Cheap, and not where the problem is.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a heavily fragmented table, to show which term became dominant — not measurements. Compare with the ordering in Physical Data Layout: on a well-sized table the top two rows are near zero and the fourth row is the whole picture.
Choosing a target, and the writers that fight you
The right target is a range chosen against the engine, the format and the row width, and no honest lesson states a number for it — a value that is right for one engine and row shape is wrong for the next, and a learner who memorises one will apply it where it is false. What is stable is how to reason about the walls on either side.
The lower wall is per-file overhead: a file must hold enough rows that reading them dominates finding them. The upper wall is parallelism and blast radius: a file should be small enough that several tasks can share the work — through row-group-level splitting if the format allows it — and small enough that rewriting or losing one is not an event.
The writers that produce the problem are all doing something reasonable. A streaming sink commits often because that is what makes data fresh. A batch job writes one file per task because that is what makes writing parallel. An ingestion connector writes one file per page because that is what makes retries simple. None of them is wrong; none of them is thinking about the reader.
The table has too many files. Which lever is the right one, given who is producing them?
when A batch job writes one small file per task and its output is much smaller than its input.
cost A redistribution of data before the write, and reduced write parallelism. Cheapest fix available and it prevents the problem instead of repairing it (Partitions: the Unit of Parallelism).
when A streaming sink commits far more often than any consumer needs data to arrive.
cost Directly reduces freshness. Only correct once you have asked what decision the freshness actually drives, and usually the answer permits a longer interval than the one configured (Cost vs Freshness).
when The producer genuinely must write small and often, and the reader genuinely needs large files.
cost A second job to operate, compute to rewrite data that was already correct, and a window where readers and the rewriter interact (File Compaction).
when File count is high because each partition holds too little data to fill a file.
cost Coarser pruning — queries touching one narrow value now read more. This is the file-size problem and the cardinality problem being the same problem (Partition Cardinality).
when The table is queried rarely, planning cost is not visible to anyone, and the compaction job would run more often than the queries do.
cost None today. Revisit when object count starts growing faster than the data does, which is the signal that the writer, not the volume, is the driver.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A streaming sink has been running for months at a short commit interval. | Queries over recent data are slow; queries over old, compacted data are fine. | The uncompacted tail holds most of the table's objects and almost none of its bytes. | Compact the tail on a schedule and keep the commit interval; measure objects per partition rather than bytes. |
| A batch job is deployed with much higher parallelism than its output size warrants. | Downstream jobs that read its output slow down, though the job itself got faster. | One output file per task. Fragmentation was exported to every consumer. | Coalesce output partitions before the write; alert on files-written-per-run in the producing job. |
| A partition column with many distinct values is added. | Total size unchanged, object count multiplied, every query slower including ones that use the new column. | Each partition now holds too little data to justify a file, so the table pays partition overhead and file overhead at once. | Revert to a coarser partition key and express the fine-grained filter through sort order instead (Clustering and Sort Order). |
| Compaction is scheduled less frequently than the fragmentation rate. | File count trends upward across weeks despite a green maintenance job. | Compaction is a rate, not a state. It is losing while succeeding. | Alert on the derivative — file count growth per day — not on the compaction job's exit status (The Pipeline Succeeded. The Data Is Wrong.). |
| A planning burst issues a very large number of storage requests at once. | Intermittently slow or failing queries with retryable storage errors, uncorrelated with data volume. | Request rate limits, hit during file discovery rather than during data transfer. | Reduce object count; and until then, reduce planning concurrency so the burst is spread out (Retry Storms: The Load You Generated Yourself). |
How to build it
Most important first.
- Set an explicit target file size and treat it as a property of the table rather than an accident of the writer. The target is a range, not a value, because a writer that must hit an exact size has to buffer unboundedly (File Compaction).
- Control the writer's parallelism at the point of write. Coalescing output partitions before writing is the cheapest possible fix and it costs a redistribution of data the job was about to write anyway (Narrow and Wide Transformations).
- For streaming, decouple commit interval from file size: commit small for freshness, then compact on a schedule. Trying to make the streaming writer produce large files directly means buffering, which means the latency you were streaming to avoid (Cost vs Freshness).
- Size partitions so that each holds enough data to justify at least one substantial file. If a partition cannot fill a file, the partitioning scheme is producing objects rather than pruning opportunities (Partition Cardinality).
- Monitor file count per partition as a first-class metric, and alert on its rate of growth. This is a failure that arrives gradually and is obvious only in hindsight (Pipeline Metrics).
- Resist the reflex to "just make files enormous". Very large files reduce parallelism, make a single corrupted object expensive, and make any rewrite of a small correction rewrite everything around it.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- File size guarantees nothing about correctness. A million tiny files and a hundred large ones hold the same rows and produce the same query results — this is a pure cost and latency property, which is exactly why it goes unnoticed for so long.
- It does not guarantee parallelism either way. Small files permit many tasks and do not make them productive; large files permit few tasks unless the format is splittable at row-group granularity (Parquet).
- Nothing in a file format or an object store enforces a size. A format specifies the layout inside a file; the writer chooses when to close one, and no consumer is protected from that choice (Open Table Formats).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a layout audit: for every partition written in the last period, record file count and total bytes, and alert when file count per partition or median file size leaves a stated band. Run it against the metadata, not by listing storage, or the check becomes the problem it is measuring.
- A complementary check on the write side: alert when a single job run produces more files than a threshold for the rows it wrote. This catches a parallelism misconfiguration on the day it ships rather than a quarter later.
- What both miss is everything about the data. A perfectly sized table can be missing a day, and a fragmented one can be flawless. They also miss the case where file count is stable because compaction is keeping up while ingest quietly doubled — the ratio is healthy right up until it is not (Volume Anomalies).
- File size and freshness are in direct tension, and this is the single clearest instance of it in the domain. A writer produces a file when it closes one, so the freshest data arrives in the smallest files.
- A platform can have both only by having two physical regions: an uncompacted recent tail optimised for arrival latency, and a compacted history optimised for reading. What it cannot have is one region that is simultaneously fresh and well-sized (File Compaction).
- Consumers should be told which side of that boundary they are querying. A query over the last hour is planning against many small files and will behave differently from the same query over last month, and someone will report that as a bug.
- Nothing here is a schema concern, and that is worth saying explicitly: file size can be changed at any time by rewriting, with no consumer visibility and no compatibility question.
- The one indirect coupling is through partitioning. Adding a partition column is a schema-level decision that immediately multiplies file count, so a change that looks like modelling is in fact a file-size change (The Partitioning Decision).
- Adding a very wide column changes how many rows fit in a target-sized file, which changes row-group granularity and therefore how selective statistics are. Size targets stated in bytes survive that; targets stated in rows do not.
- Fully recoverable by rewriting, because no information is lost. That is the good news and it is why this failure, despite being extremely common, is rarely serious if caught.
- The rewrite is not free and it is not instantaneous. Compacting a badly fragmented table means reading every object once, which is the expensive operation you were trying to avoid — paid once, deliberately, instead of on every query (File Compaction).
- Do the rewrite into new files and swap the metadata rather than deleting and re-creating, so readers never observe a table that is missing part of itself (Atomic Publish).
What can go wrong
- Compaction that cannot keep up with ingest, so file count grows despite a maintenance job that appears healthy in every dashboard.
- A fix applied to the compaction schedule while the writer keeps producing tiny files, which treats the symptom forever instead of the source.
- Over-correction into very few very large files, which caps parallelism and turns a planning problem into a straggler problem (Straggler Tasks).
- Listing throttled by the object store during a planning burst, surfacing as intermittent slow queries rather than as an obvious storage error (Saturation: The Reading Utilization Cannot Give You).
- A table format's own metadata fragmenting in parallel with the data, so that reading the manifest becomes its own small-files problem (Open Table Formats).
- "The table is small, so layout does not matter." Object count and byte count are different quantities. A table of a few gigabytes spread over a million objects is a layout emergency and a rounding error in storage terms.
- "Compaction fixed it." Compaction fixes the accumulated backlog. If the writer still produces tiny files, the table returns to its previous state at exactly the rate it did before (File Compaction).
- "One file per partition is ideal." That is fine only if a partition's worth of data is a sensible file size. Below that it wastes a partition; above it, it caps parallelism at the number of partitions a query touches.
- "Bigger is always better." Very large files reduce parallelism, make row-group-level skipping the only remaining granularity, and make a corrupted object expensive. The target is a range with two walls, not a direction (Every Optimization Buys Something and Sells Something).
- "This is an object-storage problem." It is worse there because of per-request pricing and listing semantics, but file count also drives inode pressure, directory-entry lookups and open/close overhead on a filesystem (File Systems: From Path to Blocks, Inodes).
Operating it
- File count and median file size per partition, sampled from table metadata on a schedule. The single most predictive layout metric there is.
- Query planning time separated from execution time. When planning grows as a share of total, the cause is almost always file discovery (Pipeline Observability).
- Request counts against the storage prefix, if the provider exposes them. This is the metric that makes the per-request argument concrete to someone looking at a byte-based chart (Object Storage).
- Files written per job run, emitted by the writer itself, so a parallelism regression is attributable to a deploy ("What Changed?" — Deploy Markers and the Invisible Deploys).
- At 10x ingest rate with an unchanged commit interval, file count grows roughly 10x and planning cost follows. The data volume is the least interesting part of that sentence.
- At 100x, listing-based file discovery stops being viable and a manifest-based table format becomes close to mandatory — not for transactions, but because reading one metadata file beats listing a directory tree with millions of entries (Open Table Formats).
- The pathology also arrives from cardinality with no change in volume at all. Adding one partition column with a hundred distinct values multiplies object count by up to a hundred while the bytes stay exactly the same.
- Request count is the driver, and it scales with object count rather than with data volume. This is the reason a tiny table can be expensive and the reason byte-based dashboards do not show it (What Actually Drives Data Platform Cost).
- Wasted compute is the second driver: task scheduling overhead for tasks that do almost no work, paid on every job that reads the table (Compute Waste).
- Compaction is the cost of the fix — reading and rewriting data that was already correct. It is real, it is bounded, and it is paid once per compaction rather than once per query.
- There is a metadata-storage cost too, small in bytes and significant in latency, because the metadata is on the critical path of every query while the data is not (Scan Cost).
- Larger files cost write-time buffering, which costs freshness. Every step toward better query performance here is a step away from data arriving sooner, and the platform has to pick a point on that line per table rather than once globally.
- Compaction buys query performance with compute, new file versions and an extra job to operate. On a table queried rarely, that job costs more than it saves.
- A strict size target means the writer must sometimes hold data back or redistribute it before writing, adding a shuffle to a job that did not need one (The Shuffle).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- CLOUD-SPECIFICOn object storage the cost of a file is dominated by per-request charges, listing pagination and request rate limits, so the pathology is about object count. On a local filesystem the same fragmentation costs directory-entry lookups, inode pressure and open/close syscalls instead — same direction, different mechanism and different limits.
- ENGINE-SPECIFICEngines differ in whether they list storage or read a manifest to discover files, and in whether they cache that between queries. An engine with a warm metadata cache hides small-file cost on repeat queries and exposes it fully on the first one, which makes the problem look intermittent.
- FORMAT-SPECIFICSplittable columnar formats let one large file be read by several tasks at row-group granularity, so the upper wall on file size is softer for Parquet and ORC than for a gzipped text file, which cannot be split at all and must be read by exactly one task.
- SIMPLIFIEDThe file counts in this lesson describe an illustrative dataset written two ways to make the ratio visible. They are not measurements of any system, and no target size is stated because the right one depends on the engine, the format, the row width and the query pattern.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why one unit of work per file is the natural scheduling choice for a distributed engine, and what a scheduler does when it is handed far more units than it has capacity for.