Dictionary, Run-Length, Delta and Bit Packing
Four type-aware encodings, what redundancy each one exploits, and the column property — cardinality, sortedness, range — that decides whether it does anything at all.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Which property of a column decides whether an encoding shrinks it, and how would I know which encoding my writer actually chose?
The query engine, first: some encodings let it evaluate a predicate or a GROUP BY against encoded values without materialising the originals, which is the difference between a fast aggregation and a slow one. Then the platform engineer explaining why two tables of similar row count have very different scan costs.
One page — a bounded run of values from a single column, typically thousands of them. Encodings are chosen per page, not per column and not per file, so one column can be dictionary-encoded in one page and plain in the next depending on what the writer saw.
Treat encodings as an internal detail of the format. Write Parquet, let the library decide, and never look at the footer. This is correct advice most of the time — the defaults are good and hand-tuning encodings is rarely where the win is.
A column of timestamps stored as strings gets no delta encoding at all, because delta operates on integers. The same information as a native timestamp encodes far more compactly, and the difference came from a cast nobody reviewed (Nullability & Defaults).
- A column of timestamps stored as strings gets no delta encoding at all, because delta operates on integers. The same information as a native timestamp encodes far more compactly, and the difference came from a cast nobody reviewed (Nullability & Defaults).
- A dictionary grows past the writer's page-level threshold and the encoder silently falls back to plain encoding for the rest of the column. The first pages are compact, the later ones are not, and nothing warns you.
- A
statuscolumn with five values is stored in a table sorted byorder_id, so identical statuses are scattered and run-length encoding has runs of length one. The redundancy is present in the column and unavailable to the encoder (Clustering and Sort Order). - Someone adds a monotonically increasing
ingested_attimestamp to every row at millisecond precision. It looks free — one small column — and it defeats delta encoding's best case by making every delta different. - A
GROUP BY countrythat could have run on dictionary indices runs on materialised strings instead, because a join earlier in the plan forced the column to be decoded (Columnar Execution).
What is actually happening
- Dictionary encoding replaces each value with an index into a table of distinct values. The column becomes a small dictionary plus an array of narrow integers. Its effectiveness is governed by one number: distinct values relative to total values. A
countrycolumn is nearly free; asession_idcolumn is worse than plain, because the dictionary is as large as the data and the indices are pure overhead (Cardinality: The Label That Took Down Monitoring). - Run-length encoding replaces a run of identical adjacent values with the value and a count. It is governed by *adjacency*, not by cardinality — five distinct statuses in random order have no runs, and the same five sorted have five runs. This is why sorting is the single highest-leverage encoding decision available (Partitioning).
- Delta encoding stores the difference between consecutive values instead of the values themselves. It is for ordered numeric and temporal columns: an ascending id or a sequence of timestamps becomes a series of small numbers, which then bit-pack tightly. It does nothing for an unordered column and nothing for a string.
- Bit packing stores integers in the minimum number of bits their observed range requires rather than in a fixed 32 or 64. It is the finisher applied on top of the others: dictionary indices, run lengths and deltas are all small integers, and bit packing is what turns "small integer" into "few bits" (Bits, Bytes and Words — and Why "Word" Is Not a Fixed Size).
- These compose. The common case in a real Parquet page is dictionary encoding producing indices, run-length-plus-bit-packing applied to those indices, and a general-purpose codec applied to the result. Three layers, each exploiting a different kind of redundancy, and only the last one is what most people mean by "compression".
- The property that makes these more than size reductions is that some of them are evaluable in place. A predicate
country = 'DE'can be answered by findingDEin the dictionary once and then comparing narrow integers, and a run-length-encoded run can be skipped or accepted wholesale. That is a CPU saving on top of the I/O saving, and it is why the encoding layer is not interchangeable with the codec layer (Vectorized Execution).
Four encodings and the column property each one needs
Every encoding is a bet about a specific kind of redundancy. Read the table below as a diagnostic rather than a menu: given a column, its cardinality, its sortedness and its numeric range tell you which encodings can possibly help, and none of the others will regardless of configuration.
The "degenerate case" column is the one worth memorising, because it is the case you will actually meet. Every encoding has a shape of data where it does nothing, and two of them have a shape where they make the file larger.
| Encoding | Exploits | Effective when | Degenerate case | Composes with |
|---|---|---|---|---|
| Dictionary | A small set of distinct values repeated many times | Cardinality is low relative to row count — country, status, currency, device_type | Near-unique values: the dictionary is as big as the column and the indices are pure added overhead, so writers fall back to plain | Bit packing and run-length on the resulting indices |
| Run-length | Adjacent identical values | The column is sorted or naturally clustered — a partition column, a status after a sort, a constant per file | Identical values scattered rather than adjacent: runs of length one, so the encoding stores a count for every single value | Applied to raw values or to dictionary indices |
| Delta | Small differences between consecutive values | Ordered numerics and timestamps — an ascending id, a monotonic event time, a sequence number | Unordered or randomly distributed numerics, where deltas are as large and as varied as the values themselves | Bit packing on the resulting deltas |
| Bit packing | Values that need far fewer bits than their declared width | Any small-range integer stream, which after the three above is almost everything | A column that genuinely uses its full 64-bit range, such as a hash | The finisher on dictionary indices, run lengths and deltas |
Dictionary encoding, concretely
Dictionary encoding is the one worth working through by hand, because its behaviour is entirely determined by a ratio you can compute with a single query, and because it is the encoding that also changes how a query executes rather than only how large a file is.
The dictionary is stored once per column chunk. The data becomes indices into it. If the query engine supports it, WHERE country = 'DE' is answered by locating DE in the dictionary once, then comparing small integers — and GROUP BY country can aggregate on indices and translate back at the very end. That is why dictionary encoding shows up in execution discussions and not only in storage ones (Columnar Execution).
The failure case is worth stating precisely rather than as a warning. On a column where nearly every value is distinct, the dictionary contains a copy of the entire column *and* you additionally store one index per row. Writers detect this while building the page and fall back — but the fallback is per page, so a column can be encoded one way at the start of a file and another way later, which is exactly the kind of thing that makes a size chart confusing.
1-- Run this against the source before arguing about codecs.2-- The ratio, not the row count, decides what the encoder can do.3 4SELECT5 'country' AS column_name,6 COUNT(*) AS n_rows,7 COUNT(DISTINCT country) AS n_distinct,8 COUNT(DISTINCT country) * 1.0 / COUNT(*) AS distinct_ratio9FROM events10WHERE event_date = DATE '2026-08-25'11 12UNION ALL13 14SELECT 'session_id', COUNT(*), COUNT(DISTINCT session_id),15 COUNT(DISTINCT session_id) * 1.0 / COUNT(*)16FROM events17WHERE event_date = DATE '2026-08-25';18 19-- distinct_ratio near 0 -> dictionary is close to free20-- distinct_ratio near 1 -> dictionary is overhead; the writer will fall back21-- to plain, and no codec setting will change thatWhat to notice: this is a property of *your* data, measured, not a benchmark borrowed from a format's documentation. Two tables with identical row counts can sit at opposite ends of this ratio.
country column, 12 values, 3 distinct
PLAIN
"DE" "DE" "FR" "DE" "IT" "FR" "DE" "DE" "DE" "IT" "FR" "DE"
-> 12 variable-length strings, each stored in full
DICTIONARY
dict : [0]="DE" [1]="FR" [2]="IT" <- stored once per chunk
data : 0 0 1 0 2 1 0 0 0 2 1 0 <- one small int per row
-> then BIT PACKED: 3 distinct values need 2 bits, not 32
DICTIONARY + SORTED BY country
dict : [0]="DE" [1]="FR" [2]="IT"
data : 0 0 0 0 0 0 1 1 1 2 2 2
-> then RUN-LENGTH: (0 x6) (1 x3) (2 x3) <- three pairs, not twelve ints
Same values. The third arrangement is the second one plus a sort,
and the sort is the part that costs a shuffle at write time.Sortedness is the multiplier, and it is not free
The pre block above contains the whole argument for clustering. Nothing about the data changed between the second and third arrangement — the same twelve values, the same three distinct — and the encoding got fundamentally cheaper because identical values became adjacent.
It is not free. Producing that order requires a global or per-file sort, which on a distributed writer means a shuffle: data moves across the network so that like values end up in the same place. On the write path that is the single most expensive operation available, and it is why platforms commonly write unsorted and sort later in a compaction job (File Compaction).
The second reason to sort is larger than the compression one and often gets forgotten in the discussion. Sorting makes per-chunk min/max statistics selective: in sorted data a chunk's range is narrow and disjoint from its neighbours, so a predicate eliminates most chunks without reading them. In random data every chunk's range spans nearly the full domain and the statistics eliminate nothing (The Parquet Read Path).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Uncompressed-to-compressed ratio per column chunk, tracked per run | The physical redundancy of each column is stable over time. | A type change to string, a precision increase, a cardinality explosion, a writer falling back to plain encoding. | Any change that preserves the ratio while changing meaning — ISO country codes replaced by full names compress the same and break every join. |
| Approximate distinct count per low-cardinality column, on a sample | The columns we treat as categorical still have a bounded domain. | Free text arriving in a status field, an enum gaining unbounded values, an upstream that started appending an identifier to a label. | Cardinality that is stable but wrong — a fixed set of five values where two of them now mean something different (Semantic Changes). |
| Row groups skipped versus row groups read, per query, from engine statistics | The sort order we chose is actually eliminating work for real predicates. | A compaction job that stopped sorting, a new predicate column nobody clustered on, files written by a second writer that ignores the sort. | Queries that never filter at all, which read everything by definition and will look like a pruning failure when they are simply a full scan. |
| Encoding list per column chunk compared with the previous run | The writer is still making the same physical decisions it made yesterday. | A library upgrade changing defaults, a threshold crossed mid-column, a config drift between two pipelines writing the same table. | Everything about correctness. This check is purely physical and a file can be perfectly encoded and completely wrong. |
None of these is a correctness check. They belong in the cost and layout conversation, and treating a healthy encoding report as evidence the data is right is exactly the confusion this domain exists to remove (The Pipeline Succeeded. The Data Is Wrong.).
Append files as batches arrive, in whatever order the source produced them, then raise the compression level when storage grows. Encodings still apply per page but runs are short, and every column chunk's min/max for `country` spans the whole alphabet.
Append fast and unsorted on the hot path so freshness is unaffected. A scheduled compaction rewrites closed partitions sorted by the column that appears in predicates, producing long runs and disjoint per-chunk ranges.
The two goals — fast arrival and effective skipping — are in direct conflict on the same write, because sorting requires a shuffle and a shuffle cannot be done incrementally per micro-batch. Separating them in time gives each one the path it needs, at the cost of recent data pruning worse than older data, which is a limitation you can state to consumers rather than a bug.
How to build it
Most important first.
- Give columns their real types. A date stored as a string, a number stored as a string, a boolean stored as
'Y'/'N'— each of these removes the encoding that was designed for it, and the fix is a cast at the staging boundary rather than a codec setting (SQL Transformations). - Sort or cluster on the low-cardinality column you filter on. That single choice converts dictionary encoding into run-length encoding and simultaneously makes chunk statistics selective — one decision, two independent wins (Clustering and Sort Order).
- Keep cardinality in mind when you add a column. A high-cardinality identifier added to a wide fact table is a permanent, uncompressible tax on every scan of that table, and if it is only needed occasionally it belongs in a side table.
- Reduce precision where the business does not need it. Truncating an
ingested_atto the second rather than the microsecond creates runs where there were none, and nobody downstream can tell the difference — but confirm that with the consumer rather than assuming it (Who Actually Consumes This Data). - Read the footer before tuning anything. Every Parquet and ORC writer records the encodings it used and the compressed and uncompressed size per column chunk. That is a measurement of your data rather than a benchmark of someone else's (Parquet Internals).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- All four encodings are lossless and fully reversible. A reader reconstructs the exact original values, including the ones a human would consider redundant.
- The writer chooses encodings; the reader must support all of them. This means encoding choice is never a compatibility contract with consumers — but it *is* a compatibility contract with reader library versions, since a newly-added encoding is unreadable by older readers.
- No encoding guarantees a size reduction. Dictionary encoding on a unique column reliably produces more bytes than plain, which is why writers implement a fallback rather than trusting the encoding.
- Predicate evaluation against encoded values is an engine capability, not a format guarantee. Two engines reading the same file can differ completely in whether they decode before filtering (Query Engines).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Track compressed and uncompressed size per column from the file metadata, as a scheduled job over a sample of files. The column that stops compressing is a signal that something changed upstream — a type became a string, precision increased, or cardinality exploded.
- Pair it with a distinct-count check on the columns you rely on being low-cardinality. A
statuscolumn that gains a thousand values because a source started emitting free text is a data-quality incident that first shows up as a size anomaly (Distribution Tests). - These miss anything that keeps size and cardinality stable while changing meaning — a
countrycolumn that switched from ISO codes to full names has the same cardinality, encodes identically and breaks every join (Semantic Changes).
- Encoding is a write-time cost. A writer that gathers statistics to choose an encoding per page does more work per batch, which lengthens the publish step slightly and pushes out visibility for a streaming writer.
- The sorting that makes encodings effective is the expensive part, not the encoding: sorting requires a shuffle, and a shuffle on the write path is what turns a two-minute append into a much longer job (The Shuffle).
- For that reason many platforms write unsorted on the hot path and sort during a scheduled compaction, accepting that recent data prunes badly and older data prunes well (File Compaction).
- Adding a column adds its own encoding behaviour and nothing else; existing files are untouched and readers handle mixed sets by treating the missing column as null (Schema Evolution).
- Changing a column's type changes which encodings apply, and this is where a "harmless" widening quietly costs money: an
int32widened toint64doubles the plain width and changes what bit packing can do, across every file written from then on. - Changing precision or units — seconds to milliseconds, euros to cents — has no schema signature at all and can transform a well-encoded column into a poorly-encoded one overnight. Size metrics catch it; type checks do not (Semantic Changes).
- Re-encoding is a rewrite of the affected files with a new writer configuration or a new sort order. It is a physical operation with no schema consequence, so it can be done partition by partition without coordinating with consumers.
- Sort-order changes are the risky variety, because a partial rewrite leaves some partitions sorted and some not, and any consumer that assumed ordering breaks. Nothing in a file format promises row order to a reader, so a consumer relying on it was already wrong — but they will still be broken (Reprocessing vs Retrying).
- Validate the rewrite on row count, a summed measure and a distinct count of the key columns before swapping. A re-encode should be a byte-for-byte semantic no-op and it is worth proving it (Validating a Backfill Before You Publish).
What can go wrong
- A dictionary exceeding the writer's size threshold mid-column, so later pages fall back to plain encoding and the column's footprint grows non-linearly with no warning.
- A numeric or temporal column arriving as a string, which removes delta and bit packing entirely and is invisible unless somebody looks at the types (CDC and Schema Drift).
- Sorting chosen for compression rather than for pruning, producing excellent footer statistics on a column nobody filters on.
- An engine that decodes dictionary columns before filtering, so the format's in-place evaluation advantage is unavailable regardless of how the file was written.
- A nullable column with a high null fraction where the definition-level overhead is a larger cost than anyone expects, because nulls are still recorded per value.
- "Dictionary encoding always helps." It helps in inverse proportion to cardinality and actively hurts on a unique column. Writers fall back to plain precisely because the encoding has a losing case.
- "Run-length encoding works on repeated values." On *adjacent* repeated values. A column with five distinct values in random order has no runs at all, and this distinction is the whole reason sort order is a first-class layout decision.
- "The codec is the compression." The codec is the last and usually smallest layer. Most of the reduction in a well-written columnar file happened before any general-purpose compressor ran (Why Analytical Data Compresses).
- "Encodings are an implementation detail I can ignore." True until a column stops compressing, at which point the footer is the only place the answer lives and reading it is a five-minute job.
Operating it
- Per-column compressed and uncompressed bytes from the file footer, tracked over time. The single most informative view of a table's physical health.
- Approximate distinct count per column on a sample, so a cardinality change is visible before it shows up as a cost increase (Cardinality: The Label That Took Down Monitoring).
- The encoding list the writer recorded per column chunk — a column whose encoding changed between two runs is telling you something changed upstream.
- Bytes scanned per query for the tables where you changed sort order, to confirm the change bought pruning and not just a smaller file (Scan Cost).
- At 10x rows with the same domains, dictionary and run-length encoding get relatively *more* effective: the dictionary is fixed-size and the indices grow, so its overhead amortises.
- At 10x cardinality, they get worse in exactly the same proportion, and a column that was nearly free becomes one of the largest in the table.
- At 100x, per-page decisions matter more than per-file ones, because the writer is making thousands of independent encoding choices and a threshold set for a small table behaves differently on a large one.
- Bytes retained follows encoding effectiveness directly, and it is the smallest of the cost lines.
- Bytes scanned follows encoding only for the columns a query actually reads. Column pruning happens first and dominates (Projection Pushdown).
- CPU on the write side rises with statistics gathering and, far more, with any sort required to create runs.
- CPU on the read side *falls* when an engine can evaluate predicates and grouping against dictionary indices instead of materialised strings — one of the few places in this domain where a size reduction and a CPU reduction point the same way.
- Sorting to create runs costs a write-side shuffle and locks the table into one ordering. You can have precise skipping on one column, not on five (Clustering and Sort Order).
- Reducing timestamp precision to create runs is a data decision disguised as a physical one. It is free until the day someone needs sub-second ordering, and then it is unrecoverable.
- Letting the writer choose everything is the right default and it means you cannot explain a size regression without reading the footer. That is a fair trade, but only if someone actually reads the footer.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALDictionary, run-length, delta and bit packing are the standard vocabulary of columnar storage and appear under these names in Parquet, ORC, Arrow and most analytical database engines. What differs is which combinations a writer will select and at what thresholds.
- FORMAT-SPECIFICParquet defines encodings per page with a dictionary fallback; ORC uses per-stream encodings with its own direct and dictionary variants and a separate present-bit stream for nulls. The same column can therefore encode differently and to different effect in the two formats.
- ENGINE-SPECIFICWhether a predicate is evaluated against dictionary indices or against decoded values is the engine's choice, not the file's. DuckDB, ClickHouse, Trino and Spark differ here, so the CPU advantage of an encoding is not portable across engines.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the library-version pinning that decides whether a newly-permitted encoding is readable by every consumer in the estate — an encoding upgrade is a dependency rollout, not a data change.