Chunking Pipelines
Chunking is not preprocessing and not a hyperparameter. It is the grain declaration for the retrieval index, and a boundary in the wrong place is the same class of error as a wrong fact-table grain.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does one row of your retrieval index represent — and which questions does that choice quietly make unanswerable?
The retriever, which can only ever return whole chunks; the agent, whose context window is filled with them; the reader following a citation, who needs enough surrounding text for it to be checkable; and the evaluation harness, whose recall figures are defined over these units and change meaning the moment the units do (RAG Evaluation).
One chunk is one row of the retrieval index and the atomic unit of everything downstream: retrieval returns chunks, reranking orders chunks, the context window is filled with chunks, and a citation points at a chunk. Choosing a chunk is choosing a grain, with exactly the consequences a fact table has — the unit decides which questions are answerable and which are answerable-looking and wrong (Grain: What Does One Row Represent?).
Split every document into fixed windows of a set character count with a fixed overlap. It is one line of code, it is deterministic, it never throws, and it produces a corpus of uniform well-behaved rows. For long prose of consistent structure it genuinely works, and it is a reasonable default to start from rather than a mistake to apologise for.
A table is cut in the middle. The header row lands in one chunk and half the data rows in another. Neither chunk can answer a question about the table, and both retrieve well, because both are full of the topic words.
- A table is cut in the middle. The header row lands in one chunk and half the data rows in another. Neither chunk can answer a question about the table, and both retrieve well, because both are full of the topic words.
- A numbered procedure is split at step four. The retrieved chunk begins mid-procedure with no indication that earlier steps exist, and an answer built from it confidently starts in the middle.
- A pronoun crosses a boundary. The chunk reads "it must be renewed within thirty days" and the noun it refers to is in the previous chunk — grammatical, topical, and unusable.
- A short policy that would fit comfortably in one chunk is split anyway, because the window is fixed. The corpus now holds three partial statements of a rule and no complete one.
- A source file is chunked by character count, cutting through a function body, so retrieval returns half an implementation and a closing brace.
- Overlap is raised to compensate for all of the above. Now the same sentence exists in several chunks, retrieval returns three near-identical rows, and a bounded context window holds one paragraph three times (Duplicate Rows).
What is actually happening
- Retrieval operates on whole chunks. Whatever a chunk contains is exactly what can be returned — no more, because there is no mechanism to widen it at query time, and no less, because there is no mechanism to narrow it. The chunk is the resolution limit of the entire system.
- An embedding of a chunk is a single point standing in for all the text in it. A long chunk spanning three topics has a vector that sits between them and is close to none; a short chunk covering half a thought has a vector that is precise about something incomplete. That tension is the whole chunk-size argument and it has no universal answer (Embeddings).
- Boundary placement is a separate question from size, and the more important one. A boundary at a section break loses nothing; a boundary mid-sentence damages both sides. Two chunkers with identical average sizes can differ enormously here, which is why size alone is a poor description of a strategy.
- Structure-aware chunking uses the document's own boundaries — headings, sections, list items, code blocks, table rows — which is the concrete reason the extraction stage should persist *structure* and not only text (The LLM Data Pipeline).
- Parent-child chunking separates the unit that is searched from the unit that is returned: index a small precise chunk, return the section it belongs to. This is the same move as indexing a narrow key and projecting a wide row, and it decouples two requirements that were never actually the same one (Index Types: B-tree, Hash, Partial, Expression, Covering, Full-Text).
- Overlap is a hedge against bad boundaries rather than a strategy in its own right. It buys tolerance and pays in duplicated text, duplicated vectors and duplicated retrieved rows — a standing cost on every request in exchange for reducing the frequency of one failure.
Chunking is a grain decision, so write the grain down
The table below is the same device this domain uses for fact tables, applied to a retrieval index — and the point is that it applies without modification. Each row is a legitimate choice of what one row of the index represents; each has a class of question it answers well and a class it makes structurally impossible.
Read the middle column as a declaration you would have to defend in review. "One chunk is one section of one policy document at one version" is a grain statement. "About a thousand characters" is not — it does not identify a unit, so nobody downstream can reason about whether a count of chunks means anything.
The breaksIf column carries the teaching. Notice that none of these failures is a retrieval failure in the usual sense: in every one, the search worked, the row came back, the score was high, and the row was the wrong unit for the question. That is exactly what a wrong fact-table grain does to a revenue number (Grain: What Does One Row Represent?).
| Stage | One row is | Breaks if |
|---|---|---|
| Whole document | One document at one version, embedded as a single point. | The document covers more than one topic. The vector sits between all of them and is near none, so a precise question retrieves it weakly or not at all — and if it does come back, it consumes the entire context budget. |
| Page | One printed page, as the extractor reconstructed it. | The document has any structure other than pagination. A page boundary is a property of the printer, not of the meaning, and it falls in the middle of arguments as often as between them. |
| Section or subsection | One heading and everything under it, at one version. | The extractor was wrong about what a heading is, or sections are wildly uneven — a single section running to thirty pages becomes one enormous unretrievable row. |
| Paragraph | One paragraph of one section of one document. | Meaning spans paragraphs. Conditions stated in one paragraph and exceptions in the next retrieve independently, and returning only the first produces a confidently incomplete rule. |
| Fixed-size window | A span of N characters or tokens, with boundaries decided by counting. | Anything structured is in the corpus. Tables, lists and code are cut wherever the count runs out, and the resulting halves both retrieve well and neither answers. |
| Sentence | One sentence. | Almost always. Sentences are precise and context-free in the worst sense: pronouns, ellipsis and cross-references make most of them meaningless in isolation, and retrieval returns them individually. |
| Table row | One row of one table, with its header repeated into every row. | The question is about the table as a whole ("which plan has the longest notice period"), which no single row can answer even though every relevant row retrieves. |
| Parent-child | Two units at once: a small indexed unit for search, and a larger parent returned as context. | The parent link is not maintained. A re-chunk that rewrites children without rewriting parents leaves retrieved children pointing at text that no longer corresponds to them. |
Eight strategies, eight different grains, and one property shared by all of them: the failure never looks like a failure. Retrieval succeeds, the score is high, the text is on topic, and the unit is wrong for the question being asked.
Where the boundary falls, made concrete
Averages hide this completely, so here is one document, one table and one question. The table is small, unremarkable, and exactly the kind of content people ask an assistant about. A fixed window that does not know the table exists cuts it in a place that is fine for prose and destructive here.
Follow what the retrieved chunk contains after the cut. It has the plan name, it has the number, and it does not have the column headers — so the number is present without the thing that says what the number means. The model is now guessing at a schema, and a guess about which column a value belongs to is not something a confidence score will reveal.
The structure-aware version below is not cleverer. It simply refuses to place a boundary inside a table, which is a rule you can state in one sentence and assert in a test. Most of the value in chunking comes from a handful of rules of exactly that shape (Data Tests).
Split at every N characters with an overlap of M, applied uniformly to Markdown, HTML, PDF extractions and source files alike. Tables, lists, code blocks and procedures are cut wherever the counter runs out.
Split at section boundaries. If a section exceeds the size budget, split it at paragraph boundaries within the section, and never inside a table, a list or a code block. If a table must be split, repeat its header into each part and mark the chunk as a fragment.
Retrieval can only return whole chunks, so a chunk is only useful if it is self-contained for some class of question. Structural boundaries are the document author's own statement about where self-contained units begin and end, and a character counter has no access to that information at all. This is the same reason a fact table is grained on a business event rather than on a convenient row count.
Extraction output, with structure preserved
──────────────────────────────────────────────────────────────
## 4.2 Notice periods
| Plan | Notice | Refund |
|------------|---------|--------------|
| Monthly | 7 days | pro-rata |
| Annual | 30 days | none |
| Enterprise | 90 days | by contract |
## 4.3 Reactivation
...
Fixed-size window, no structure awareness
──────────────────────────────────────────────────────────────
chunk_017 "## 4.2 Notice periods | Plan | Notice | Refund |
| Monthly | 7 days | pro-rata | | Annual | 30"
chunk_018 "days | none | | Enterprise | 90 days | by contract |
## 4.3 Reactivation ..."
Question: "How much notice does an Enterprise plan need?"
chunk_018 scores highly: it contains "Enterprise", "90 days",
"by contract" — and no header row. Nothing in the chunk says
which column "90 days" sits in. The answer is produced anyway.
chunk_017 also scores highly: it contains the header row and
the heading, and the Enterprise row is not in it.
Structure-aware chunking, boundaries at section edges
──────────────────────────────────────────────────────────────
chunk_017 whole of section 4.2: heading, header row, all 3 rows
chunk_018 whole of section 4.3
Same question retrieves one row that contains the header, the
Enterprise row, and the other two rows for comparison.Choosing a strategy, and what each one costs
There is no winning option here, which is why this is a decision table rather than a recommendation. What makes the choice tractable is that it is not one choice: it is one choice per source class, and most corpora have three or four classes with obviously different natural units.
The criterion that decides most of these is not retrieval quality in the abstract. It is the question shape: whether the questions people actually ask are answered by a fragment, by a section, or by comparing across a whole document. A corpus of "what is the rule for X" questions wants sections; a corpus of "which of these is largest" questions wants something no chunking strategy provides, and needs a structured table rather than a retrieval index at all.
Notice the last option. When the questions are aggregations over structured content, the honest answer is that the data should be in a table and queried with SQL, and forcing it through a retrieval index is an architecture mistake that no chunk size can repair (OLTP vs OLAP).
What shape of question do people actually ask of this content, and what is the smallest self-contained unit that answers it?
when The documents have real headings and sections that were written to be self-contained — policies, manuals, reference documentation, knowledge-base articles.
cost Depends entirely on extraction quality, and produces wildly uneven chunk sizes that need a size fallback inside long sections. Buys boundaries that lose nothing.
when Long unstructured prose, transcripts without turn markers, or a corpus so heterogeneous that no structure signal is reliable. Also the correct first implementation while you learn the corpus.
cost Cuts structured content indiscriminately, and the overlap that mitigates that duplicates text into every retrieval. Buys predictability and one line of code.
when Precision and completeness genuinely conflict: long structured documents where a paragraph identifies the right place but a section is needed to answer.
cost Two units to keep consistent, a second lookup on the retrieval path, and a re-chunk that must rewrite both halves together or leave dangling parents. Buys precise search and complete answers at once.
when Prose with topic shifts that do not align with any markup, and a corpus large enough that boundary quality is worth real complexity.
cost The boundary rule becomes a dependency with its own version, and a change to it re-chunks everything. Non-obvious to test, and harder to explain to whoever is debugging retrieval at 2am.
when The content is already structured — tickets, product records, table rows, log entries. One record is one row and there is nothing to split.
cost Needs the record boundary and its metadata from the source system, and answers no question that spans records. Buys an exact grain for free.
when The questions are aggregations, comparisons or filters over structured data — counts, maxima, "which of these", "how many since".
cost Requires a real table and a query path beside the retrieval one, and a router that decides which to use. Buys correct answers to a class of question that nearest-neighbour search cannot answer at any chunk size (Federated Query).
Reproducible chunking: identity, not position
Everything above is undermined if a re-run produces different rows for unchanged input. Chunking has to be a deterministic function of the extraction output and the chunker version, and the chunk id has to be a function of what the chunk *is* rather than where it happened to fall in an ordering.
Position-based ids fail in a specific and expensive way. Insert a paragraph near the top of a long document and every subsequent chunk shifts by one; with ordinal ids, every one of them is a new row, so the whole tail is re-embedded despite being byte-identical. With content-based ids, only the chunks that actually changed get new ids and the rest are recognised as already present (Upserts and Merges).
Putting the chunker version inside the id is the second half of the discipline. It means a strategy change writes into a disjoint id space, so the old and new grains cannot silently occupy the same index — the failure this whole lesson is about becomes structurally impossible rather than merely discouraged (Idempotent Data Pipelines).
The technique for keeping boundaries stable under edits, when boundaries cannot follow structure, is content-defined chunking: choose split points where a rolling hash over a sliding window meets a condition, so a boundary depends on nearby bytes rather than on distance from the start of the file. An insertion then shifts only the boundaries near it (Rolling Hash (Polynomial Hashing)).
1CHUNKER_VERSION = 'struct-v3' # bump this and the whole corpus gets new ids2 3 4def chunk_id(doc_id, doc_version, start, end, text):5 """Identity of a chunk = what it is, not where it fell in an ordering."""6 payload = chr(31).join([7 doc_id, # which document8 doc_version, # which version of that document9 CHUNKER_VERSION, # which rules produced this boundary10 str(start), str(end),11 sha256(text), # what the chunk actually says12 ])13 return sha256(payload)[:32]14 15 16# The write is then an upsert keyed on (chunk_id, embedding_model_version),17# never an insert. Re-running an unchanged document is a no-op; re-running a18# document whose third paragraph changed rewrites the chunks covering that19# paragraph and leaves the rest of the corpus untouched.Three properties fall out, and none of them is about hashing. A re-run of an unchanged document produces identical ids, so the index write is idempotent rather than an append. A chunker change produces a disjoint id space, so two grains cannot coexist unnoticed in one index. And because the text hash is part of the id, a chunk whose bytes changed gets a new id even when its offsets did not — the exact case a position-only id gets wrong.
How to build it
Most important first.
- Chunk on the document's own structure first, and fall back to size only *within* a structural unit. A section that fits becomes one chunk; a section that does not is split at paragraph boundaries inside it, never across it.
- Never separate a table, a code block or a list from its header. If a split is unavoidable, repeat the header into every piece and record on the chunk that you did, so a reader can tell a repeated header from an original one.
- Prepend a small deterministic context header to every chunk — document title, section path, effective date — so a retrieved fragment carries where it came from rather than arriving as anonymous text (Context Construction & Grounding).
- Write the grain down in a sentence: "one chunk is one subsection of one policy document at one version". If that sentence cannot be written, retrieval measurements are not comparable across time and should not be plotted on one axis (Dataset Documentation).
- Use parent-child where precision and completeness genuinely conflict — long structured documents, legal text, reference manuals — and skip it where they do not, because it doubles the units you have to keep consistent.
- Version the chunker and put the version inside the chunk id, so a strategy change produces a disjoint id space and cannot silently mix grains inside one index (Vector Data Engineering).
- Choose per source class rather than globally. Policy documents, chat transcripts, code and support tickets have different natural units, and a single setting is a compromise that is wrong for most of them.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Chunking guarantees a complete cover: every byte of the cleaned document belongs to at least one chunk. That is worth asserting explicitly, because the signature failure of a structure-aware chunker is quietly dropping text that fell outside every recognised structure.
- It guarantees addressability when offsets are recorded: each chunk resolves to a byte range in a stored document version, which is what makes a citation verifiable rather than decorative (Citations).
- It guarantees nothing about semantic completeness. There is no property of any chunker that ensures a chunk contains enough context to answer anything, and no test that can be run against the chunker alone to establish it.
- With overlap enabled it explicitly does not guarantee that a sentence appears once. Deduplicating what retrieval returns becomes the retriever's responsibility, and a retriever that does not do it wastes context on repetition (Deduplication).
- Grain stability holds only within a chunker version. Across versions, chunk counts, chunk ids and every retrieval metric defined over them are incomparable, and reporting them on one time series is a measurement error rather than a trend.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check with the best return is a boundary integrity assertion expressed against the extraction structure rather than the flat text: no chunk begins or ends inside a table, a code block or a sentence, and no table row is separated from its header (Data Tests).
- It misses semantic incompleteness entirely. A chunk can start and end on immaculate sentence boundaries and still be unusable because its subject is a pronoun resolved two paragraphs earlier.
- The complement is a retrieval-side assertion over a curated set: for each known question and answer, the answer text must be contained *within a single retrieved chunk*. That measures precisely what the boundary test cannot, and it requires a dataset to exist before it can be run (Golden Datasets).
- Chunking itself adds no meaningful latency — it is cheap, deterministic and CPU-bound. What it adds is the cost of *changing your mind*: a chunking change is a full-corpus rebuild, which is the slowest operation the platform has (Re-embedding).
- How much of a document must be reprocessed when a little of it changes is decided by the chunker. Size-based boundaries shift every offset after an edit, so a one-paragraph change re-chunks and re-embeds the whole document; structure-aware boundaries localise the change to one section.
- That property is simultaneously a freshness property and a cost property, and it is the strongest practical argument for boundaries that follow content rather than character counts (Cost vs Freshness).
- A chunker change is a grain change, and therefore a breaking change to every consumer at once: retrieval metrics, evaluation baselines, cached results, and any analysis of which chunks get retrieved (Breaking Schema Changes).
- The safe pattern is the one any grain change to a fact table uses — build the new grain beside the old, evaluate both against the same questions, switch, then retire. Never mutate in place, because a half-re-chunked index is a corpus at two grains (Atomic Publish).
- Chunk *metadata* evolves far more gently. Adding a section-path field to new chunks is backward compatible; making retrieval depend on that field is not, until the whole corpus carries it (Backward Compatibility).
- Re-chunking runs from the persisted extraction dataset. It costs chunking compute and a full re-embed of whatever changed, but no re-parsing and no access to the source systems (Reprocessing vs Retrying).
- Because the chunker version is inside the chunk id, a re-chunk writes a disjoint set of ids rather than overwriting the previous ones. Rollback becomes a matter of pointing retrieval back at the old version, and cleanup becomes a deliberate delete rather than an accident (Rolling Back Data).
- Recovering from a bad boundary rule without re-embedding is not possible, because the vector is a function of the chunk text. This is the clearest case in the module of a cheap decision with an expensive undo (Re-embedding).
What can go wrong
- A structure-aware chunker silently dropping text that belonged to no recognised structure — the preamble before the first heading is the classic one, and it is often the summary.
- Overlap set high enough that retrieval returns several copies of one passage and the context budget is consumed by redundancy (Token Budgets).
- A chunker whose boundaries depend on a library's tokenizer, so a routine dependency upgrade changes every boundary in the corpus without a line of your own code changing.
- Two chunkers coexisting in one index after a partial re-chunk, so retrieval mixes grains and no measurement over the index means anything (Grain: What Does One Row Represent?).
- Chunk ids derived from ordinal position, so inserting a paragraph at the top of a document renumbers everything below and forces a re-embed of an unchanged tail.
- The mitigation failing: a boundary-integrity test written against cleaned flat text, which cannot see the table structure that only ever existed in the extraction output.
- "Chunk size is a hyperparameter to tune." It is a grain declaration. Changing it changes what one row means, which invalidates every prior measurement rather than improving on it (Grain: What Does One Row Represent?).
- "Overlap fixes bad boundaries." It reduces how often a boundary destroys an answer, and it guarantees duplication on every request. That is a mitigation with a permanent cost, not a fix.
- "Smaller chunks are more precise, so smaller is better." Smaller chunks are more precise about less. A chunk that retrieves perfectly and contains half the rule is worse than a vaguer chunk containing all of it.
- "Chunking is preprocessing, so we can change it any time." It is the most expensive change in the pipeline: the vectors are a function of the chunk text, so every one of them has to be recomputed (Re-embedding).
- A chunk inherits its parent document's classification and nothing enforces that inheritance unless the classification is copied onto the chunk row at build time (Data Classification).
- Redaction has to happen before chunking, or it has to be reapplied to every chunk. A redaction applied to the source document after chunks exist does not reach them, and the chunk table becomes the unredacted copy (Data Masking, Tokenisation & Encryption).
- A chunk with no resolvable parent is undeletable in practice: a deletion request against a document cannot find rows that do not point back at it, which is a second reason offsets and document ids are not optional (Deletion Requests).
Operating it
- Chunk length distribution per source class, with both tails inspected. A hard spike at exactly the maximum size means the structural path is never being taken and the fallback is doing all the work (Distribution Tests).
- Share of chunks that begin mid-sentence, computed structurally rather than by punctuation heuristics. It should be near zero and in a first implementation it never is.
- Overlap in retrieved results: how much of the text handed to the model is duplicated across the returned rows. This is the direct measurement of what overlap costs you per request.
- Chunks per document version over time, so a chunker or tokenizer change appears as a step in a chart rather than as a complaint about answer quality three weeks later (Volume Anomalies).
- At ten times the corpus, chunk-size choices that were merely suboptimal become the dominant storage line, and parent-child starts paying for itself by keeping the *indexed* unit small while the returned unit stays useful.
- At a hundred times, re-chunking stops being something you do casually. It becomes a scheduled migration with a rollback plan and a validation gate, which promotes chunking from a tuning knob to an architectural commitment (Planning a Backfill).
- Source heterogeneity scales worse than volume does. One chunker across five document classes is a compromise that is wrong for four of them, and per-class chunkers multiply the version bookkeeping this module keeps insisting on.
- Chunk count is the multiplier on every downstream cost at once: vectors to compute, vectors to store, index size, and rows competing for a bounded context window. A smaller average chunk moves all four in the same direction (What Actually Drives Data Platform Cost).
- Overlap is pure duplication. It adds vectors and bytes that carry no information the corpus did not already have, in exchange for tolerance of bad boundaries.
- Chunking compute itself is close to free. The expense is that every chunking decision has to be paid for again at the embedding stage, across the whole corpus, every time it changes (Compute Waste).
- Structure-aware chunking produces better boundaries and inherits an extraction quality it does not control. When the parser is wrong about what a heading is, the chunker is confidently wrong about sections; fixed windows are uniformly mediocre and never surprising.
- Small chunks retrieve precisely and answer incompletely. Large chunks answer completely and retrieve vaguely. Parent-child buys both and costs a second lookup, a second stored unit, and a retrieval path with more to go wrong in it.
- Per-source-class chunkers fit each corpus better and multiply the number of versions in flight. Every version is a column, a validation gate and a rollback target, and there is a real point where the bookkeeping costs more than the fit is worth.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat the chunk is the resolution limit of retrieval, and that the boundary decision is a grain decision, holds for any store and any embedding model. What differs between systems is only whether the store can hold the parent text alongside the indexed child, which decides whether parent-child needs a second lookup.
- SOURCE-SPECIFICThe natural unit is a property of the document class. A policy manual has sections that are self-contained by design; a chat transcript has turns that are meaningless alone and coherent in runs; source code has functions whose meaning depends on imports far above them. A single strategy across all three is wrong for at least two.
- SIMPLIFIEDPresented as one chunker producing one index. Production retrieval commonly indexes the same corpus at several grains at once and merges results, which does not remove the grain question — it means several grains have to be declared, versioned and measured separately instead of one.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what deterministic identity buys you when the same document is processed twice by two workers that cannot see each other — content-addressed ids are the reason that race is harmless rather than a duplicate.