ORC
A sibling columnar design with the same goals and different specifics: stripes instead of row groups, row-index strides for finer skipping, and row-level ACID in the ecosystem it grew up in.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
ORC and Parquet solve the same problem — so what is actually different, and when does the difference decide anything?
A query engine reading a Hive-managed table, and the platform team that inherited one. Also anybody evaluating formats who has been told "they are basically the same" and needs to know which parts of that are true.
One stripe — a horizontal slice of rows, ORC's analogue of a Parquet row group. Within a stripe, each column is a set of streams, and every few thousand rows a row-index entry records statistics for that stride, which gives a finer default skipping unit than Parquet's row group.
Pick whichever format your engine's documentation used in its examples. This is a reasonable heuristic and produces the right answer most of the time, because ecosystem fit matters more than the format's intrinsic properties.
A team standardises on Parquet across the platform and inherits a Hive warehouse of ORC tables with row-level updates. Converting them means giving up the ACID behaviour those tables relied on, which nobody costed (Open Table Formats).
- A team standardises on Parquet across the platform and inherits a Hive warehouse of ORC tables with row-level updates. Converting them means giving up the ACID behaviour those tables relied on, which nobody costed (Open Table Formats).
- The reverse: a team writes ORC because the cluster did, then adopts an engine whose ORC reader supports fewer of the format's pruning features than its Parquet reader, and loses skipping without changing any data.
- Bloom filters are enabled on a column in ORC and the engine in use ignores them, so the write cost was paid and the read benefit never arrived (Data Engineering Anti-Patterns).
- Delta files from row-level updates accumulate because compaction is not running, and read performance degrades in a way that looks like a data-volume problem and is not (File Compaction).
- Both formats are used in the same lake for the same logical table, so every consumer needs both readers and any layout audit has to handle two vocabularies (Data Platform Anti-Patterns).
What is actually happening
- ORC is columnar with a horizontal cut, exactly like Parquet. The cut is called a stripe; within a stripe each column is stored as a set of streams — a present-bit stream for nulls, plus data streams whose shape depends on the column's type and encoding.
- The null representation differs and it matters. Parquet encodes nulls with definition levels per value; ORC uses a separate present-bit stream per column. For a sparse column those two choices produce different overheads, which is one of the few places where the physical difference between the formats is genuinely visible (Parquet Internals).
- ORC keeps statistics at three levels: file, stripe, and every N rows within a stripe (the row-index stride, conventionally ten thousand rows). Parquet's standard granularity is the row group, with page-level indexes as an optional extra. So ORC's default skipping unit is finer, and that is a real difference rather than a marketing one (Predicate Pushdown).
- Both formats put their metadata at the end and both are splittable at their horizontal cut. Both support the same family of encodings — dictionary, run-length, delta-style integer encodings — and both apply a general-purpose codec on top (Dictionary, Run-Length, Delta and Bit Packing).
- The distinctive ecosystem feature is row-level ACID in Hive: updates and deletes are written as delta files alongside a base file, and a reader merges them at read time. A compaction periodically folds deltas into a new base. This is the same mechanism modern table formats use, arrived at earlier and scoped to one ecosystem (Open Table Formats).
- The honest summary of the difference is: same architecture, different granularity defaults, different null encoding, different ecosystem history. Neither is a successor to the other and both are actively maintained (Parquet).
The same architecture in a different vocabulary
Almost everything true of Parquet's structure is true of ORC's with the words changed. Both cut horizontally then store column-major within the cut. Both put a map at the end. Both keep per-unit statistics so a reader can skip. Both are splittable at the horizontal cut and both compress inside it.
The table below is the translation, plus a column for where the mapping actually breaks down. That last column is the point: an equivalence table with no differs entries would be teaching that the formats are interchangeable, and in the three places noted they are not.
The one to remember is granularity. ORC's row-index stride gives a finer default skipping unit than Parquet's row group. Parquet can reach comparable granularity with page-level indexes where the writer emits them and the engine reads them, which is more conditions than "by default".
| Concept | Parquet | ORC | Where the mapping breaks |
|---|---|---|---|
| Horizontal cut | Row group | Stripe | Same idea, different default sizes and different tuning parameters. A stripe size copied from a row group setting is not a like-for-like configuration. |
| Per-column storage inside the cut | Column chunk of pages | Set of streams per column | ORC splits a column into several typed streams (present bits, lengths, data); Parquet keeps one chunk of pages. Footprint for sparse and variable-length columns differs as a result. |
| Nulls | Definition levels per value | A separate present-bit stream per column | For a highly sparse column the overheads are genuinely different, and this is the clearest physical difference between the two formats. |
| Statistics granularity | File, row group; page indexes optional | File, stripe, and every N rows (row-index stride) | ORC's finer granularity is the default; Parquet's equivalent depends on the writer emitting page indexes and the engine reading them. |
| Optional pruning structures | Bloom filters, column and offset indexes | Bloom filters, row-index entries | Both are optional to write and optional to consume. Engine support differs per format, so presence in the file does not imply use at read time. |
| Row-level update support | None — a file format only | Base plus delta files, merged at read, compacted periodically (in Hive) | This is not a format feature you can carry elsewhere. Outside its metastore, ORC delta files are just files with no transaction interpreting them. |
| Nested data | Definition and repetition levels | Nested types with length streams for lists | Both support nesting; statistics quality on nested leaves varies by writer and engine in both, and is not comparable across them. |
Row-level updates, and what they cost
The feature ORC is most associated with is not a storage-layout property at all: in the Hive ecosystem it supports row-level inserts, updates and deletes on a columnar table, which is otherwise a contradiction in terms — columnar files are immutable and rewriting one to change a row is the whole problem.
The mechanism is the one every modern table format converged on. A base file holds the bulk. Updates and deletes are appended as delta files. A reader merges base and deltas at read time to produce the current state. A background compaction periodically folds deltas into a new base and drops them (Upserts and Merges).
The operational consequence is what teams meet in production: read cost grows with delta volume, so a table that is updated frequently and compacted rarely gets progressively slower with no change in data volume. That degradation looks exactly like growth and is not, which is why the delta count is a metric worth having on a dashboard rather than in a runbook (Pipeline Metrics).
- 1Write base
Writes the bulk of the partition as immutable columnar files.
guarantees The base is complete and immutable as of its transaction; readers see all of it or none of it.
fails by Being large enough that any correction requiring a base rewrite is expensive, which is exactly why deltas exist.
- 2Apply update as delta
Appends a small delta file recording the changed and deleted rows, with their positions or keys.
guarantees The change is durable and visible to readers at the next transaction boundary, without rewriting the base.
fails by Accumulating. Each delta is cheap to write and permanently adds work to every subsequent read until compaction.
- 3Read-time merge
Reads base plus every delta and reconciles them into current rows.
guarantees Readers see a consistent current state as of a transaction, not a mixture of pre- and post-update rows.
fails by Degrading proportionally to delta count, which presents as gradually slower queries with no obvious cause (Stale Dashboards).
- 4Compaction
Folds deltas into a new base file and retires the old base and deltas.
guarantees Read cost returns to base-only, and the merged result is equivalent to the pre-compaction merged view.
fails by Not running. This is the characteristic operational failure — it is a background job, so its absence is silent (File Compaction).
- 5Snapshot expiry
Retires old bases and deltas that no reader or time-travel window still needs.
guarantees Storage is reclaimed and only reachable versions are retained.
fails by Expiring a version a long-running read or a recovery still needed, turning a maintenance job into a data-loss event (Data Retention).
The same five stages describe every modern open table format. ORC-on-Hive arrived at them first and scoped to one ecosystem; the pattern is what transfers, not the implementation (Open Table Formats).
ACID table support, compaction scheduling, delta file layout and the exact set of statistics and index structures written by ORC libraries have all changed across Hive and ORC versions, and engine support for each differs. Treat the base-plus-delta pattern and the three statistics levels as the stable ideas and verify current documentation for any specific behaviour.
How to actually choose
There is no benchmark that settles this, and any that claims to is measuring one engine, one data shape and one query mix. The decision is an ecosystem decision, and framing it that way makes it answerable in an afternoon instead of a quarter.
The compare below is the argument in its usable form. Notice that neither side is "faster" — the reason to prefer one is about who can read your data, what your platform already runs, and which failure modes your team is equipped to operate.
Run a comparison on a sample dataset with one engine, pick the winner, and standardise on it. The result is a real measurement of one configuration and it does not predict behaviour under a different engine, a different data shape, a different sort order or a different query mix.
List the engines and languages that must read this data and check reader maturity for each format. Check what the rest of the platform uses. Decide whether you need row-level updates and whether you are prepared to operate compaction. Then pick, and write down why.
The intrinsic differences between the two formats — stride granularity, null encoding — are smaller than the variation between engines reading the same file, and far smaller than the effect of partitioning and sort order. Reader maturity and platform consistency are stable properties you can verify; a benchmark result is a measurement of a configuration you are about to change (Benchmark Fallacies: Confident Numbers That Are Wrong).
How to build it
Most important first.
- Choose by ecosystem fit first. Which engines read your data, which of them have the better-maintained reader for each format, and what does the rest of your platform already use? That answer dominates every intrinsic property (Choosing an Analytical Platform).
- Do not mix formats for the same logical table. The cost of two vocabularies in every audit, every migration and every consumer's dependency list exceeds any difference between the formats (Data Platform Anti-Patterns).
- If you are on ORC, tune the row-index stride and the stripe size the way you would tune Parquet's row group — from predicate selectivity, verified in a query profile (The Parquet Read Path).
- If you use ORC's row-level ACID, treat delta compaction as a first-class scheduled job with monitoring, not as a background nicety. Uncompacted deltas are the characteristic operational failure of that feature (File Compaction).
- Before enabling bloom filters on a column, confirm your engines actually use them. It is a write-side cost with an engine-dependent read-side benefit (Query Engines).
- For a new platform with no existing commitment, Parquet's breadth of reader support across engines and languages is the practical tiebreaker — which is an ecosystem argument, not a technical superiority claim.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Statistics at file, stripe and stride level are sound bounds, exactly as in Parquet: a unit is skipped only when no value in it can match.
- Splittability is guaranteed at stripe boundaries, so parallel reads work without an external index.
- The file is self-describing — schema and metadata live in the footer — so a reader needs nothing external to interpret it.
- Row-level ACID guarantees are Hive's, not the file format's: the format provides base and delta files, and the transactional semantics come from the metastore and the reader that merges them. Moving those files to a different engine does not carry the guarantees with them.
- Nothing guarantees an engine uses every pruning structure present. Bloom filters and stride indexes are optional to consume, and a file can carry them for a reader that ignores them.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check specific to ORC-with-ACID is a delta file count and age assertion per partition: alert when deltas accumulate beyond a threshold or when the oldest delta exceeds the compaction interval.
- Alongside it, the same layout audit as for Parquet — stripe count and size, statistics coverage, schema fingerprint per file (Data Tests).
- Neither says anything about correctness. A partition with perfectly compacted deltas and healthy stripes can be missing a day, and the audit will report it as excellent (The Pipeline Succeeded. The Data Is Wrong.).
- Stripe size is the same freshness lever as a Parquet row group: a writer buffers a stripe before flushing, so larger stripes mean later visibility.
- ORC's row-level updates give something Parquet files alone do not — the ability to apply a correction without rewriting the base file, which shortens the path from "we found the bug" to "the table is right" (Upserts and Merges).
- That freshness is borrowed. Every delta file makes reads more expensive until compaction folds it in, so the fast correction is paid for later (Cost vs Freshness).
- Additive column changes behave as in Parquet: old files lack the column, readers project null (Backward Compatibility).
- ORC identifies columns positionally in some historical arrangements and by name in others depending on writer and reader configuration, which makes column reordering a genuinely riskier operation than it is in Parquet. Verify the behaviour for your engine before relying on it (Breaking Schema Changes).
- Converting a table between ORC and Parquet is a full rewrite with a validation step, and it is where row-level ACID history quietly does not survive — the merged current state does, the update history does not (Reprocessing vs Retrying).
- A rewrite between formats is a read-write-validate-swap like any other layout change, and the validation must include a row count and a summed measure because a format conversion is exactly the kind of change teams assume is lossless (Validating a Backfill Before You Publish).
- For ACID tables, recovery from a bad update means restoring from a snapshot or replaying from the source; the delta mechanism is not a version history you can walk backwards indefinitely (Backfills).
- A corrupt stripe costs that stripe. Like Parquet, a lost footer costs the file, and the recovery path is re-derivation from raw (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- Delta files accumulating because compaction stopped, degrading reads progressively with no error.
- An engine with a weaker ORC reader than its Parquet reader, so identical layout decisions produce different pruning.
- Bloom filters or stride indexes written and ignored, paying write cost for no read benefit.
- Column reordering under a positional-mapping configuration, which reads successfully and returns the wrong column's values — the worst failure in this module, because it is silent and typed correctly.
- A format conversion that dropped ACID history without anyone recording that it had.
- Two formats in one lake, doubling every reader dependency and every audit path.
- "ORC is the old one." Both formats are actively maintained and both are used in current systems. Ecosystem history explains where you find each one; it does not make either obsolete.
- "ORC is faster than Parquet" (or the reverse). Neither claim survives contact with a different engine, a different data shape or a different query. The differences that are real — stride granularity, null encoding, ACID support — are specific and situational, and none of them is a general speed claim (Benchmark Fallacies: Confident Numbers That Are Wrong).
- "ORC gives us ACID." Hive gives you ACID over ORC files. Move the files elsewhere and you have base and delta files with no transactional layer interpreting them.
- "They are basically the same, so it does not matter." They are architecturally the same and operationally different. The decision is real; it is just an ecosystem decision rather than a benchmark one.
Operating it
- Delta file count and oldest delta age per ACID partition — the operational signal specific to this format (Pipeline Metrics).
- Stripes read versus skipped, and stride-level skipping where the engine reports it (Query Engines).
- Stripe size distribution per file, which reveals a writer flushing on time rather than on size.
- Format inventory across the lake: which tables are ORC, which are Parquet, and which are both (The Data Catalog).
- At 10x, the finer default stride granularity can matter for selective predicates, and stripe sizing becomes a deliberate decision rather than a default.
- At 100x, the differences between the two formats are dwarfed by partitioning, clustering and file count — the same conclusion as every other physical-layout question in this domain (Physical Data Layout).
- At 100x updates, ACID delta accumulation becomes the dominant operational concern and compaction scheduling becomes the platform's main maintenance job.
- The same drivers as Parquet: bytes fetched at column granularity, units skipped by statistics, planning cost proportional to file count (Scan Cost).
- ACID tables add a read-time merge cost proportional to delta volume, and a periodic compaction compute cost to remove it (Compute Waste).
- Optional structures — bloom filters, fine stride indexes — cost write CPU and storage and pay back only on engines that read them.
- Migration between formats costs a full rewrite of every affected byte, once, plus the validation around it.
- Finer skipping granularity costs more metadata per file and more work at write time. It pays on selective predicates and costs on full scans.
- Row-level ACID buys corrections without rewrites and costs a read-time merge plus a compaction pipeline to operate.
- Choosing the format your ecosystem already uses buys tooling, examples and working readers, and costs whatever intrinsic advantage the other one had — which is almost always the correct trade.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- FORMAT-SPECIFICStripes, streams, present-bit null encoding and row-index strides are ORC's specifics; Parquet uses row groups, pages and definition levels. The architecture is shared and the vocabulary and default granularity are not, which is why layout audits are not portable between them.
- ENGINE-SPECIFICReader maturity for ORC versus Parquet differs by engine and is the single biggest practical factor. An engine may honour stride indexes and bloom filters in one format and only file-level statistics in the other, producing different pruning from identical layout decisions.
- TOOL-SPECIFICORC's row-level ACID is a Hive metastore feature built on base and delta files, not a property of the format. Its transactional semantics, its compaction behaviour and its recovery story all belong to Hive, and none of them travels with the files.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what a transaction boundary means over files in object storage, which is the guarantee the base-plus-delta pattern is built to provide.
- — DevOps / Production Engineering owns the scheduled compaction job's delivery, alerting and rollback — the characteristic ORC-ACID failure is an unmonitored background job, which is a production-engineering problem.