Why Analytical Data Compresses
Columns of one type with repeating values compress in ways rows of mixed types cannot — and the chain from fewer bytes to a faster query has three places it can break.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Why does the same dataset shrink dramatically when stored column by column, and why does that not automatically make queries faster?
Anyone who pays for a scan and anyone who waits for one: the analyst whose dashboard reads a year of events, the Spark job that pulls files across a network, the finance team that sees a storage line grow every month, and the platform engineer asked why one table costs more to query than another the same size.
The unit here is the value run inside one column — a contiguous sequence of values of a single type, in the order they physically sit in the file. Compression works on that run. Nothing in this lesson is about a row, which is why row-oriented storage cannot reach the same result.
Turn on gzip. Storage is compressed, the files are smaller, the invoice moves, and nobody has to learn anything about encodings. For a landing zone of JSON this is genuinely a good first move, and it costs one configuration flag.
Generic compression on a row-oriented file works on a window of bytes that interleaves an integer id, a timestamp, a float and a UTF-8 country code. The repeating structure is there but it is diluted by three unrelated types between every recurrence, so the compressor finds far less to exploit than the same values grouped by column.
- Generic compression on a row-oriented file works on a window of bytes that interleaves an integer id, a timestamp, a float and a UTF-8 country code. The repeating structure is there but it is diluted by three unrelated types between every recurrence, so the compressor finds far less to exploit than the same values grouped by column.
- A gzipped CSV is not splittable. One file becomes one task regardless of how many workers are idle, so the storage saving is paid for by losing parallelism on every read (File Size and the Small-Files Problem).
- The query still reads every column. Compression reduced the bytes on disk but the reader has no way to know where
countrylives inside the stream, soSELECT countrydecompresses the whole thing (Projection Pushdown). - Decompression is CPU. On a job that was already CPU-bound on parsing and expression evaluation, a heavier codec moves the bottleneck rather than removing it (Computing or Waiting?).
- Someone concludes the format made queries fast, tunes the codec, and gets nothing — because the win came from column pruning and statistics, not from the compression algorithm (The Parquet Read Path).
What is actually happening
- Compression exploits redundancy. A column is the densest concentration of redundancy in a dataset because every value in it has the same type, the same width and the same domain —
countryholds a few hundred distinct strings,statusholds five,event_dateholds one value repeated for an entire partition. - That is why columnar layout and compression are the same subject and not two. Grouping values by column does not compress anything on its own; it puts similar values next to each other so that lightweight encodings and a general-purpose codec have something to work with (Row vs Column Storage).
- Two distinct layers are at play and they are constantly confused. Encodings — dictionary, run-length, delta, bit packing — are type-aware transforms that a reader can often evaluate without fully materialising the original values. Codecs — Snappy, zstd, gzip, LZ4 — are general byte compressors applied on top, and their output must be decompressed before anything can look at it (Dictionary, Run-Length, Delta and Bit Packing).
- The chain a format is sold on is: fewer bytes stored, so fewer bytes read from disk, so fewer bytes moved across the network, so less time spent waiting. Every arrow in that chain is conditional. Fewer bytes read only follows if the reader can locate what it needs; less waiting only follows if the job was waiting on I/O in the first place.
- The counterweight is CPU. Every byte saved is bought with decode work at read time and encode work at write time, and on modern hardware the balance frequently favours a fast codec over a small one — which is why analytical systems default to Snappy or zstd at a low level rather than to maximum compression (When the Memory Bus Is the Bottleneck).
The same values, two arrangements
Take four columns of an events table — an integer id, a date, a country code and a revenue amount — and write ten rows of it. Row-major, the bytes on disk alternate between four unrelated types, and any repetition of DE is separated from the next DE by an id, a date and a float. Column-major, every DE sits next to every other DE.
A general-purpose compressor works on a sliding window of recent bytes. In the row-major arrangement it sees a repeating *structure* and can exploit that, but the repeating *values* are far apart relative to its window. In the column-major arrangement the repeating values are adjacent, and adjacency is the entire thing compression is built to exploit.
This is also why the column layout is what makes the type-aware encodings possible at all. Run-length encoding needs a run; a run only exists if identical values are contiguous. Dictionary encoding needs a bounded domain; a domain only exists if the values in a block are all from the same column. Neither has anywhere to stand in a row-major file (Row vs Column Storage).
ROW-MAJOR (one record at a time, four types interleaved)
1001 | 2026-08-25 | DE | 41.90
1002 | 2026-08-25 | DE | 12.00
1003 | 2026-08-25 | FR | 41.90
1004 | 2026-08-25 | DE | 8.50
-> on disk: int,date,str,float,int,date,str,float,int,date,str,float ...
COLUMN-MAJOR (one column at a time, one type per run)
order_id : 1001 1002 1003 1004 <- ascending ints -> delta + bit pack
event_dt : 2026-08-25 x4 <- one value -> run-length
country : DE DE FR DE <- tiny domain -> dictionary
revenue : 41.90 12.00 41.90 8.50 <- repeated values -> dictionary
-> each run is homogeneous, so a type-aware encoding has something to exploit
BEFORE a general-purpose codec ever runsFewer bytes, and the three places the chain breaks
The argument for compression is a chain: fewer bytes stored means fewer bytes read from disk, which means fewer bytes moved over the network, which means less time waiting. Each arrow is real and each one is conditional, and knowing which condition failed is the difference between tuning the right thing and tuning a codec for a week.
The first break is locatability. Fewer bytes on disk only reduces bytes read if the reader can find the subset it needs. A gzipped CSV has fewer bytes and the reader must still stream all of them, so nothing downstream improves.
The second break is splittability. If the compressed unit is the whole file, one file is one task. The job is now bounded by its largest file rather than by its total size, and adding workers does nothing (Distributed Data Processing).
The third break is where the time actually went. If the job spent its time in a shuffle, in expression evaluation, or waiting on a skewed partition, then halving the bytes read changes a number that was never the bottleneck (Data Skew).
Projection is the largest single lever in a columnar format and costs nothing to use — it is available the moment you stop writing SELECT *.
Driven entirely by sortedness and partition key choice. On randomly ordered data min/max ranges overlap everywhere and skip almost nothing.
A function of cardinality and run length. Low-cardinality and sorted columns shrink a great deal; identifier columns barely move.
The knob people reach for first and the one with the least headroom. It trades bytes against CPU on both the write and the read side.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a typical analytical scan, shown to establish an ordering rather than a magnitude — these are not measurements. The ordering is the teaching: what you skip beats what you shrink, every time.
Cases where compression does not help
The honest version of this subject includes its exceptions, because they are common enough that a team will meet one within a year. Every row below is a real situation where a correct-sounding compression decision produced no benefit or a regression.
Notice what they have in common: in each case the bytes did get smaller. The failure is never that compression did not compress; it is that the saving was not on the path the workload was constrained by, or that it removed a property (splittability, adjacency) that mattered more than size.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A team gzips a directory of CSVs to cut storage. | Storage falls; every downstream Spark job takes about as long as before, or longer. | Gzip is not splittable, so each file became one task. Parallelism dropped by exactly as much as file count multiplied by size. | Rewrite to a splittable columnar format, or at minimum to a splittable codec, and size the files so each one is a reasonable task (File Size and the Small-Files Problem). |
A high-cardinality session_id column is added to a wide fact table. | The table's compressed footprint grows out of proportion to the number of columns added. | A dictionary of near-unique values is as large as the column, so the encoder falls back to plain and the codec finds nothing to exploit. | Accept it, or move the identifier to a narrower table joined on demand. Do not tune the codec — the column has no redundancy to find. |
| Codec changed from Snappy to a high gzip level to save storage on a hot table. | Scan cost is unchanged, write jobs take noticeably longer, and query CPU rises. | The workload was CPU-bound on decode and evaluation, not on bytes read. A heavier codec added work to the side that was already the constraint. | Reserve heavy codecs for cold data. For hot tables optimise what is skipped, not what is shrunk (Computing or Waiting?). |
| A streaming writer flushes many very small Parquet files. | Total compressed size is larger than expected and query planning is slow. | Footer, page-header and dictionary overhead are per file. Below a certain file size that overhead exceeds what encoding saves, and the reader also pays a per-file open. | Compact on a schedule and accept the freshness cost, which is the real trade being made (File Compaction). |
Data sorted by an ascending event_id rather than by the filter column. | Encodings look excellent in the file footer; queries still read every row group. | Sortedness on the wrong column compresses well and prunes nothing, because min/max on the filter column still overlaps across every chunk. | Sort or cluster on the column that appears in predicates, and accept that only one such ordering exists per table (Clustering and Sort Order). |
How to build it
Most important first.
- Store analytical data column-major before reaching for a heavier codec. Layout decides how much redundancy is available to compress; the codec only decides how much of it gets taken (Parquet).
- Choose a splittable arrangement. Parquet and ORC compress per page or per stream inside a file that remains splittable at row-group boundaries; a whole-file gzip does not, and that difference costs parallelism on every read (Physical Data Layout).
- Sort or cluster on the low-cardinality column you filter on most. Sortedness turns dictionary encoding into run-length encoding and makes per-chunk min/max statistics selective, which is where the real read saving comes from (Clustering and Sort Order).
- Default to a fast codec (Snappy, or zstd at a low level) for hot data and a heavier one for cold archival data you rarely read. That is a read-frequency decision, not a storage decision (Storage Lifecycle).
- Measure bytes scanned, not bytes stored. Storage is a small and predictable line; scanning is the large and variable one, and the two respond to different changes (Scan Cost).
- Keep very wide, sparse or high-cardinality columns in mind as the exception. A column of UUIDs, hashes or free text compresses close to not at all, and a table dominated by such columns will not behave like the examples in a format's documentation.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Compression is lossless in every format discussed in this domain. What you write is byte-identical to what you read back, including trailing whitespace, string case and float representation. Nothing here is a rounding step.
- No format guarantees a size reduction. A column of random 128-bit identifiers can encode to slightly *more* bytes than its raw form once dictionary and page overhead are counted, and this is normal rather than a bug.
- Decompression is deterministic and version-stable for the standard codecs, so a file written years ago reads identically today. Encoding *choices* are made by the writer and are not part of the reader contract — a reader must handle any encoding the format permits.
- Nothing guarantees a faster query. Fewer bytes read is a necessary condition, not a sufficient one, and a job bound on shuffle or on expression evaluation will not move (The Shuffle).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that catches a bad compression decision is a bytes-scanned-per-query metric grouped by table, watched against its own history. A jump after a layout or codec change is visible immediately and attributable.
- Pair it with a row-count reconciliation across the rewrite: any recompression or format migration must produce the same row count and the same sum of a monetary column as the input (Reconciliation).
- It misses value-level corruption from a bad writer implementation, because counts and sums can both survive a mangled string column. It also misses the case where the query got cheaper because a filter was silently dropped rather than because pruning improved.
- Compression is a write-time cost that sits on the critical path of publishing. A heavier codec makes each batch take longer to write, which pushes out the moment a consumer can see it — small on a nightly job, material on a five-minute micro-batch.
- Streaming writers face this most sharply: they must choose between small frequently-flushed files that compress poorly and larger buffered files that compress well and arrive later (Streaming Ingestion).
- Nothing about compression changes what a consumer can ask. It changes what the ask costs, which is a cost and capacity conversation rather than a freshness one.
- Codec choice is a physical property with no schema consequence — you can rewrite a table from gzip to zstd and no consumer's SQL changes. Say that out loud, because it is one of the few genuinely reversible decisions in this domain.
- Encoding choice is likewise the writer's business. A reader that understands the format understands all of its encodings, so a writer upgrading from plain to dictionary encoding breaks nobody.
- What does break is a reader pinned to an old library version meeting a newly-permitted encoding or codec. That is a dependency-compatibility problem, not a schema one, and it is the reason platform teams pin codec choices centrally rather than per pipeline.
- Recompressing a dataset is a rewrite: read every file, write new ones, swap the pointer. It is idempotent if the output location is derived from the input and the swap is atomic, and it is a data-loss event if you rewrite in place and the job dies halfway (Atomic Publish).
- Always validate the rewrite before publishing — row count, a summed measure, and a spot-check of the widest string column — because a codec migration is exactly the kind of change everyone assumes is safe (Validating a Backfill Before You Publish).
- A corrupt compressed block is unrecoverable from the file itself. Recovery means re-deriving the file from raw, which is one more reason the raw landing zone exists (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- A whole-file gzip that removes splittability, so one large file pins one worker and the job's runtime is set by a single task (Straggler Tasks).
- A heavier codec chosen to reduce storage on a table that is scanned constantly, shifting cost from the small line to the large one.
- High-cardinality columns defeating every encoding, so a table that "should" compress well does not, and the team spends a week tuning the codec instead of looking at the column.
- Very small files, where per-file and per-page overhead dominates and the compressed set is larger than the uncompressed one would have been in bigger files (File Compaction).
- A reader library too old for the encoding the writer chose, which surfaces as an unreadable file rather than as a slow one.
- "Parquet is just compressed CSV." The compression is the least interesting part. What Parquet adds is a self-describing schema, per-chunk statistics and a layout that lets a reader skip columns and row groups without decompressing them — none of which a compressed CSV can do at any codec setting.
- "We compressed the data, so queries got faster." Only if the query was waiting on bytes. Measure where the time went before attributing it (Measure Before You Optimize).
- "Higher compression is better." Higher compression is smaller and slower on both ends. The right level is a function of how often the data is read versus how long it is kept.
- "Compression ratios from the benchmark will hold for us." They will not. Ratios are a property of your column cardinality, sortedness and value distribution, and two tables of identical size can behave completely differently (Benchmark Fallacies: Confident Numbers That Are Wrong).
Operating it
- Bytes scanned per query, by table and by user. This is the number compression is supposed to move and the only one worth defending (Scan Cost).
- Compressed versus uncompressed size per column, available from the file footer in Parquet and ORC. The column that is not shrinking is the one to look at, and it is usually an identifier or a free-text field.
- Write-side job duration before and after a codec change, so the cost that moved onto the writer is visible rather than absorbed silently.
- File count and file size distribution per partition, because a compression conversation is very often a small-files conversation wearing a different hat (File Size and the Small-Files Problem).
- At 10x volume the encodings behave the same way; what changes is that the columns that do not compress start to dominate the file and set its size.
- At 100x, the decision that matters is no longer the codec but whether a query can skip whole files and row groups. Compression reduces what you must read; layout reduces what you must consider reading, and only the second one scales (Partition Pruning).
- Higher cardinality at scale is the quiet reversal: a
user_idcolumn with a thousand distinct values encodes beautifully and the same column with a hundred million does not, so the same table gets relatively *less* compressible as the business grows.
- Storage cost falls roughly with compressed size, and it is usually the smallest line in an analytical platform.
- Scan cost falls with bytes actually read, which depends far more on column pruning and row-group skipping than on the codec (What Actually Drives Data Platform Cost).
- Network cost falls with compressed bytes moved, which matters most where compute and storage are separated and every read crosses a network (Separating Storage from Compute).
- CPU cost rises on both sides — encode on write, decode on read — and it is the line that a heavier codec silently increases.
- Smaller files cost CPU on every read forever, in exchange for a storage saving taken once. For a hot table that is often the wrong direction and for a cold archive it is clearly the right one.
- Sorting to make encodings effective costs a shuffle at write time and constrains the layout to one query pattern. It buys skipping for that pattern and nothing for the others (Clustering and Sort Order).
- Larger row groups compress better and give coarser skipping; smaller ones skip more precisely and compress less. There is no setting that wins both and the right answer depends on your predicates.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat grouping like values together exposes redundancy is a property of data, not of any product, and holds for every columnar format ever built. What varies is which encodings a given format defines and whether a reader can evaluate a predicate against the encoded form.
- FORMAT-SPECIFICParquet and ORC compress per page and per stream inside a splittable container; a gzipped CSV compresses the whole file as one stream and is therefore not splittable at all. The same codec produces completely different read behaviour depending on what wraps it.
- SCALE-SPECIFICBelow a few hundred megabytes per table the entire subject is noise and a plain uncompressed file is fine. Above the point where a single query scans more than a machine's memory, layout and skipping dominate and the codec becomes the least important knob.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a non-splittable file destroys parallel execution — the work-partitioning argument underneath it is theirs, not this domain's.