The LLM Data Pipeline
Documents to ingest to clean to chunk to metadata to embed to index to retrieval. Nine stages, nine promises, and most retrieval failures happen in the first three.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A PDF lands in a shared drive. What has to happen, in what order, before an agent can retrieve one sentence from it — and what does each of those steps actually promise?
The retrieval path, which reads this pipeline's output on every request and has no way to ask it a question; the agent, which will happily answer from whatever comes back; and the engineer who has to explain why a document everyone can see in the shared drive is invisible to the assistant (RAG Overview).
The grain changes at four separate points and each change is a place a metric or an answer can go wrong. One source file becomes one document version, becomes one text extraction, becomes many chunks, becomes one vector per chunk, becomes one row in a serving index. A count of documents and a count of index rows are different numbers by design, and confusing them is the first thing that happens on every coverage dashboard (Grain: What Does One Row Represent?).
A loader function walks a directory, reads each file, splits the text into fixed-size pieces, embeds them and writes them to a vector store, all in one script someone runs by hand. It fits on one screen, it works on the sample set, and for a proof of concept it is exactly the right amount of engineering — building an incremental, versioned, checkpointed pipeline before you know whether retrieval helps is the more expensive mistake.
A scanned contract returns an empty string from the parser. No exception is raised, no chunk is produced, the run reports success, and the document is simply absent from the corpus forever (The Pipeline Succeeded. The Data Is Wrong.).
- A scanned contract returns an empty string from the parser. No exception is raised, no chunk is produced, the run reports success, and the document is simply absent from the corpus forever (The Pipeline Succeeded. The Data Is Wrong.).
- HTML pages are ingested with their navigation, cookie banner and footer intact. That boilerplate repeats on every page, so the most consistently retrievable text in the corpus is the cookie banner, and it crowds out real content in the context window (Context Selection & Compression).
- The script is run a second time. Nothing is keyed, so every chunk is inserted again; retrieval now returns the same paragraph twice and the agent reads it as corroboration (Deduplication).
- One file fails halfway through a long run and the script exits. There is no checkpoint, so the choice is to abandon the run or restart from zero and duplicate everything already written (Checkpointing).
- No metadata was attached, so there is no way to filter retrieval by tenant, by document class, by effective date or by permission. Every one of those becomes a full rebuild rather than a query change (Metadata Filtering).
- A document in a language the splitter was not designed for is cut by character count mid-word, and the resulting chunks embed as something between two words that do not exist.
What is actually happening
- Ingestion is a fan-in of heterogeneous sources — a file share, a wiki, a ticket system, a code repository, a product database. Each has its own change signal, its own authentication, its own pagination and its own extraction problem, and a pipeline that treats them uniformly gets the change signal wrong for most of them (Ingestion Sources).
- Extraction is the lossy stage nobody budgets for. A PDF is a description of marks on a page, not a document: reading order, column boundaries, table structure and the distinction between a header and a heading are all *reconstructed* by the parser rather than read out of the file. Two parsers on the same file legitimately produce different text.
- Cleaning is a semantic decision wearing the costume of a string operation. Removing repeated boilerplate improves retrieval; removing a repeated legal footer removes the answer to the question the legal team asks most. There is no way to tell those apart without knowing the corpus.
- Chunking decides the grain of the index — what one retrievable row represents — and therefore decides which questions are answerable at all (Chunking Pipelines).
- Metadata is what makes the index a dataset rather than only a similarity search. Filters, permissions, effective dates, document class and pipeline versions are columns, and columns you did not write at build time cannot be added at query time (Metadata Filtering).
- Embedding is the only stage with a foreign, per-item dependency in the middle of a batch job. Rate limits, partial failure, retries and cost-per-item all live here, and none of them behave like the rest of a data pipeline (Embedding Pipelines).
- Retrieval is a second pipeline, running per request, that must agree with the first about two things: the embedding model version and the metadata contract. Nothing enforces that agreement unless you build the enforcement (Data Contracts).
Nine stages, and what each one promises
Writing the chain out as stages is worth doing precisely because it forces the middle column to be filled in. Most arguments about retrieval quality are really disagreements about which stage was supposed to guarantee something, and the honest answer is usually that none of them did.
Read the guarantees column top to bottom and notice how weak it is in the middle. Landing raw bytes is a strong promise. Extraction promises only that a string was produced. Cleaning promises idempotence and says nothing about whether what it removed was noise. By the time text reaches the embedder, the pipeline has made no claim at all about fidelity to the original document — and every stage after that treats the text as if it were the document.
The failsBy column is the one to keep open during an incident. Each stage fails in a characteristic way, and recognising which one you are looking at collapses the search immediately: an absent document points at enumeration or extraction, a duplicated passage points at the index write, and confident irrelevance points at a version mismatch between build and query.
- 1Discover
Enumerates source items and determines which have changed since the last successful run.
guarantees Every item present at scan time is either enumerated or recorded as skipped with a reason. Nothing about items the source did not return.
fails by Reading the first page of a paginated listing and stopping, so the corpus contains the newest documents and silently nothing else.
- 2Land raw
Copies the untouched bytes into immutable object storage, addressed by content hash.
guarantees What arrived is preserved exactly, so every later stage is re-runnable without touching the source again (The Raw Landing Zone).
fails by Writing under a mutable path, so a re-ingest overwrites the only evidence of what the previous run actually saw.
- 3Extract
Turns bytes into text plus structure: reading order, headings, tables, page boundaries.
guarantees Only that a string was produced. Not that it is the document, not that reading order is right, not that a table survived as a table.
fails by Returning an empty or near-empty string for a scanned or image-only file and exiting successfully.
- 4Clean
Removes navigation, headers, footers and repeated boilerplate; normalises encoding and whitespace.
guarantees Idempotence, if it is a pure function of the extraction output. Nothing whatsoever about whether what it removed was noise.
fails by Stripping a repeated legal footer that happened to be the answer to the question the corpus is most often asked.
- 5Chunk
Splits cleaned text into retrievable units and records each unit's offsets back into the stored document.
guarantees A complete cover of the document by addressable units, at one declared grain (Chunking Pipelines).
fails by Cutting a sentence, a table or a numbered list in half, so neither half answers the question and both retrieve well.
- 6Attach metadata
Writes source id, document version, permissions, dates, document class and every pipeline version onto each chunk.
guarantees That each chunk carries the filters retrieval will need — but only the filters someone thought of at build time (Metadata Filtering).
fails by Omitting a permission field, which converts every future access rule from a query change into a corpus rebuild.
- 7Embed
Calls an external model once per chunk and stores the resulting vector with the model version.
guarantees A vector in one model version's space. Nothing about semantic quality and nothing about comparability with another version's vectors.
fails by Stopping part-way under rate limiting, leaving a corpus where some chunks have vectors and some do not (Embedding Pipelines).
- 8Publish index
Writes vectors and chunk metadata into the serving index and makes them visible to readers.
guarantees If published as a swap, readers see the whole new corpus or the whole old one. If published by incremental writes, no such property exists (Atomic Publish).
fails by Appending on a re-run rather than upserting on a deterministic chunk id, so each run duplicates the corpus.
- 9Retrieve
Encodes the query with the pinned model version, searches, applies metadata filters, optionally reranks.
guarantees The nearest neighbours of the query vector among the vectors present, under the filters supplied (Reranking).
fails by Encoding the query with a different model version from the one that built the index — confident results, no error, no relevance (Re-embedding).
Two stages make strong promises: landing raw, and publishing atomically. Everything between them promises far less than the systems downstream assume, and the assumption is never written down anywhere a reviewer would see it.
Which document types extract cleanly, whether a scanned page yields any text without a separate recognition step, and how tables survive are properties of the specific extraction library and its version rather than of the file format — and they change between releases. Treat "we can parse PDFs" as a claim to re-test against your own corpus after every upgrade, not as a capability to assume.
Four persisted datasets, four places to re-enter
The single most consequential design choice in this pipeline is not the chunker or the model. It is whether the intermediates are written down. A pipeline that keeps raw bytes, extraction output, chunk rows and vectors has four re-entry points; a pipeline that keeps only the index has none, and every fix costs a full reprocess from the source system.
This is the same argument that puts a staging layer between raw and curated in any warehouse, and it lands harder here because the stages have such different costs. Re-parsing a corpus is CPU. Re-chunking is cheap. Re-embedding is a per-item charge to an external service. Collapsing them means every bug is priced at the most expensive stage it touches (Raw, Staging, Curated: Layers by Purpose).
The version columns are what make the re-entry points usable. "Re-run everything produced by extractor v3" is a query; "re-run everything, because we cannot tell which documents v3 touched" is a budget conversation. Storing the version that produced each row is a few bytes and it is the difference between the two.
A single job reads files, parses, cleans, chunks, embeds and writes to the index in one pass, keeping nothing in between. Fixing a chunk-boundary bug means re-reading every source document, re-parsing all of them, and paying the per-item embedding cost for the entire corpus again.
Raw bytes, extraction output, chunk rows and vectors are each written down with the version of the code that produced them. A chunking change re-runs from extraction. A parser fix re-runs from raw for the affected file types only. A new embedding model re-runs from chunks.
The stages differ by orders of magnitude in cost and by orders of magnitude in how often they change. Persisting between them turns "reprocess the corpus" from one undifferentiated expensive operation into the cheapest re-run that can actually fix the bug in front of you — which is the same reason a warehouse keeps a staging layer it never serves (Full Refresh vs Incremental).
The checks that make the pipeline auditable, and what each one still misses
Every check below is cheap, and every one of them has a blind spot large enough to drive an incident through. That is the normal condition of data quality and the reason a portfolio beats any single test — but it is sharper here, because the consumer of this dataset is a model that cannot notice that something is missing (Data Quality).
Note which check catches the pipeline's signature failure. Extraction plausibility is the one that finds the empty parse, and it is almost never present in a first implementation, because the run was green and the vector store had rows in it.
The last row is a different kind of check and worth calling out. A retrieval smoke set is a liveness probe, not a quality measure: it answers "can the corpus still find the things it could find yesterday", which is exactly the question a publish should be gated on. Measuring whether retrieval is *good* is a separate pipeline with a separate dataset (Evaluation Data Pipelines).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Extraction produced text above a length floor, with a prose-like ratio of alphabetic characters, per document. | The parser produced something that is plausibly a document rather than nothing. | Scanned or image-only files, password-protected documents, a parser that returned an empty string, a truncated download. | Extraction that is long and wrong — interleaved columns, a table flattened into a run of digits, ligature garbage. Length says nothing at all about readability. |
| Chunks per document compared against the same document at its previous version. | This document decomposed the way it did last time. | A parser upgrade that changed structure, a cleaning rule that ate a section, a source that started exporting a different HTML skeleton. | A document whose content genuinely changed by that much, which produces an identical signal and needs a human to adjudicate. |
| Every chunk offset resolves to a byte range in the stored raw document at the recorded version. | Each chunk is attributable to a real span of a real document, so a citation is checkable (Citations). | Chunks produced from a stale extraction, chunks left behind after a document was re-landed, and any chunk that was written by hand. | A correct offset into a correctly stored document that was itself parsed wrongly — the pointer is perfectly honest about the wrong text. |
| Vector count equals chunk count for the current embedding model version. | The embedding stage completed across the whole corpus at one version. | Partial re-embeds, rate-limited runs that gave up quietly, chunks added after the last embed job ran (Embedding Pipelines). | Vectors that exist but were computed from different chunk text — a count matches whether or not the contents correspond, which is why the check needs a content hash to be worth much (Reconciliation). |
| Retrieval smoke set: a fixed list of questions, each with a document that must appear in the top results. | The corpus can still find things it could find before this publish. | A failed index swap, a metadata filter that excludes everything, an embedding version mismatch between build and query. | Degradation on every question not in the smoke set, which is nearly all of them. It is a liveness probe and treating it as a quality metric is how a suite stays green through a regression. |
Four of the five compare two counts. That is not a coincidence: counting is the cheapest way to observe a pipeline whose output no human reads, and it is also why every row's blind spot is a value-level error that preserves the count.
How to build it
Most important first.
- Land the raw bytes first, immutably, under a content hash, and never parse from the source again. Everything downstream becomes re-runnable and the source system stops being in the critical path of every experiment (The Raw Landing Zone).
- Persist the extraction output as its own dataset. It is the most expensive stage and the one you will re-run chunking against dozens of times; keeping it turns a chunking experiment into a cheap job instead of a full re-parse (Raw, Staging, Curated: Layers by Purpose).
- Make chunk ids a deterministic function of document version, chunker version and byte offset. That single decision converts every re-run from an append into an idempotent upsert and removes the whole duplicate class of failures (Idempotent Data Pipelines, Upserts and Merges).
- Attach metadata at chunk time from the document record, never inferred at query time from the text. A permission derived by pattern-matching the chunk body is a permission that will be wrong under pressure (Data Access Control).
- Fail loudly on an empty or implausibly short extraction, and route the document to a dead-letter dataset with the reason. Silence is the failure mode this pipeline is most prone to (A Dead-Letter Queue Is a Workflow, Not a Bin).
- Publish the index by swap or by keyed upsert, never by unkeyed append, so a re-run cannot double the corpus and a half-finished run cannot be read (Atomic Publish).
- Pin the embedding model version in one place that both the build job and the query encoder read. Two configuration files that happen to agree today is not a pin (Vector Data Engineering).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The pipeline guarantees that every source item it enumerated and successfully extracted is represented in the index. It guarantees nothing about items it failed to enumerate, and an enumeration bug is invisible from inside the pipeline (Missing Rows).
- Chunk-level addressability is a guarantee you can actually make and should: every chunk resolves to a byte range of a stored document version, which is what makes a citation checkable rather than decorative (Citations).
- Delivery into the index is at-least-once unless chunk ids are deterministic and the write is an upsert. With those two properties the *state* of the index is effectively-once regardless of how many times the job ran — a property of the sink design, not of the pipeline (Upserts and Merges).
- Retrieval guarantees the nearest neighbours of the query vector among the vectors present, under the filters supplied. It cannot signal that the best answer was never ingested, and it will return a full page of results either way (Vector Search: Embeddings, Similarity and ANN).
- Nothing in the chain guarantees that the extracted text says what the document says. That gap is the largest single source of bad answers and it is a parsing problem, not a retrieval one.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The stage-level check that earns its place first is extraction plausibility: every ingested document produced text above a length floor, with a ratio of alphabetic characters to total characters that looks like prose. It catches scanned pages, protected files and parsers that returned nothing.
- It misses extraction that is long and wrong. Interleaved columns, a table flattened into a stream of digits and ligature garbage all pass a length check comfortably, and only a human reading a sample finds them (The Dimensions of Data Quality).
- Pair it with a coverage reconciliation between the source system and the chunk table, because the plausibility check can only speak about documents that reached it — and the enumeration bug that never delivered a document is the one it structurally cannot see (Reconciliation).
- End-to-end freshness is the age of the source change, not the age of the last run. A pipeline that runs hourly over a source it only re-scans nightly is a nightly pipeline with an hourly cron on it.
- The stages have very different natural cadences: discovery can be continuous, extraction is per-changed-document, and a full re-embed is a rare, expensive, deliberate event. Forcing them onto one schedule makes the cheap stages slow or the expensive stage ruinous (Cost vs Freshness).
- Freshness is per-document, and the useful statistic is a maximum rather than a mean: the oldest un-reindexed document is the one that will embarrass you, and it disappears entirely into an average (Freshness Monitoring).
- Every stage has a version, and every version change is a semantic change to the dataset below it. A parser upgrade, a cleaning rule, a chunk size, an embedding model: none of them alters a schema and all of them alter meaning (Semantic Changes).
- Adding a metadata column is the one genuinely backward-compatible change here, and it is only useful for chunks written after it — which means a new filter usually implies a rebuild of the corpus rather than an
ALTER(Schema Evolution). - Source formats evolve underneath you. A wiki that starts exporting a different HTML structure changes the extraction output for every page at once, and the only signal is chunks-per-document moving (Volume Anomalies).
- The persisted intermediates are the recovery plan. A parser fix re-runs from raw; a chunking change re-runs from the extraction dataset; a new embedding model re-runs from the chunk table; a corrupted index rebuilds from the vector dataset. Each is the cheapest re-run that can fix its bug (Reprocessing vs Retrying).
- Re-run scoped to the affected documents, not the whole corpus. That requires knowing which documents a given version of a given stage produced, which is the reason the version columns exist (Planning a Backfill).
- Build into a new index and switch, so a failed rebuild costs storage rather than availability, and so the previous index remains a rollback target (Rolling Back Data).
What can go wrong
- Enumeration that stops early — a paginated API read to the first page only — producing a corpus that contains the newest documents and nothing else.
- A parser that returns an empty string and an exit code of zero, which is the most common single cause of "the agent does not know about that document".
- A cleaning rule tuned on one source that silently removes real content from another.
- An unkeyed insert on re-run, doubling the corpus and filling the context window with duplicates (Duplicate Rows).
- A metadata field missing, discovered only when someone asks for permission-aware retrieval and the answer is a rebuild (RAG and Agent Memory Security).
- The mitigation failing: a length-floor check calibrated on English prose that rejects every correctly extracted document in a language with a different average token length.
- "Ingestion is the easy part." Ingestion and extraction are where most retrieval failures originate, and they are the stages with the least instrumentation and the fewest tests (Data Ingestion).
- "The vector store is the pipeline." The vector store is the last stage. Everything that decides answer quality happened before the data reached it.
- "We can add metadata later." Metadata is written per chunk at build time. Adding a field later means re-chunking or re-embedding the corpus to populate it, which is a migration rather than a change (Re-embedding).
- "If retrieval returns nothing useful, improve the search." Check first whether the answer is in the index at all. A large share of "retrieval quality" investigations end at a document that never parsed (Missing Rows).
- Extraction copies document content into a second store, and that copy inherits the original's classification without inheriting its access controls. The extraction dataset is routinely the least-protected complete copy of the corpus in the company (Data Classification).
- Permission metadata must be captured at ingestion from the source system's own model. Reconstructing it later from folder paths or document text produces an access rule that is right most of the time, which is the worst possible property for an access rule (Data Access Control).
- A document withdrawn at the source has to be removed from raw, extraction, chunks, vectors and the serving index. Without deterministic ids linking those five, the removal is a manual search (Deletion Requests).
Operating it
- Documents enumerated, extracted, chunked, embedded and indexed as five counts on one chart per run. A drop between two adjacent stages localises the fault immediately, and it is the highest-value chart this pipeline has (Pipeline Metrics).
- A dead-letter dataset with one row per rejected document and a machine-readable reason, reviewed rather than merely written (Data Incidents).
- Chunks per document, distribution and outliers, per source class. Both tails are informative: a document with one chunk probably failed to parse and a document with thousands probably had its boilerplate multiplied.
- A retrieval smoke set run after every publish — a fixed handful of questions whose expected source document is known, checked for presence rather than for ranking quality (Quality Alerting).
- At ten times the corpus, the hand-run script becomes an orchestrated, checkpointed, incremental job, and the change detection that was optional becomes the whole design (Incremental Processing).
- At a hundred times, extraction and embedding stop fitting in one process and become distributed work with all the ordinary partitioning and straggler concerns — one enormous document can hold up a batch exactly the way one hot key holds up a shuffle (Data Skew, Straggler Tasks).
- Source count scales the problem faster than document count. Ten sources means ten change signals, ten auth mechanisms and ten extraction behaviours, and the pipeline's complexity tracks that number rather than the byte volume (Ingestion Sources).
- Extraction is CPU-heavy and paid once per document version; embedding is paid per chunk to an external service; index storage is paid per vector, continuously. The three scale with different things, which is why one schedule for all of them is always wrong for two of them (What Actually Drives Data Platform Cost).
- Chunk size is a cost lever with an unusual double effect: smaller chunks mean more vectors to embed and store, and also more retrieved rows competing for a bounded context window (Token Budgets).
- Persisting intermediates costs storage and saves recomputation. The trade is decided by how often you expect to change chunking and embedding, and in the first year of a corpus the answer is nearly always "more often than you think" (Compute Waste).
- Persisting four datasets instead of one costs storage and four schemas to maintain. It buys targeted reprocessing, which is the difference between a chunking experiment taking an afternoon and taking a budget approval.
- Aggressive cleaning improves retrieval on average and occasionally deletes the one paragraph that mattered. Conservative cleaning keeps everything and lets boilerplate dominate. Neither setting is safe and the choice has to be made per source (Chunking Pipelines).
- Deterministic chunk ids make re-runs idempotent and make every chunker change a corpus-wide identity change, because the id contains the chunker version. That is the correct behaviour and it means you cannot quietly slip a chunking tweak into production.
The pipeline behind a retrieval answer
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Source documents | |
|---|---|
| One record is | One document, in whatever the producer stores. |
| Promises | To be the thing the answer will eventually be attributed to. |
| Breaks when | Nobody records the version. When the document changes, the index still holds the old one and the citation points somewhere that no longer says that. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe stage sequence and the promise each stage can make hold for any retrieval corpus regardless of store, model or framework. What varies is how many stages are collapsed into one library call, which changes where you can re-enter the pipeline but not what each step can lose.
- SOURCE-SPECIFICExtraction difficulty is a property of the source format and its producer, not of the pipeline. Structured Markdown from a wiki extracts nearly losslessly; a scanned PDF may yield no text at all without a separate recognition step; a spreadsheet has no reading order to recover, so any linearisation is an invention.
- SIMPLIFIEDDrawn as a single linear chain over a single corpus. Production systems commonly run several chunking strategies over the same extraction output into separate indexes and merge results at query time, which multiplies the version bookkeeping described here rather than replacing it.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how a parser upgrade reaches production. Every stage of this pipeline has a version, and a version bump is a deployment whose blast radius is a dataset rather than a service — which means it wants a canary corpus and a rollback path, not just a passing test suite.
- — Distributed Systems owns what happens when extraction and embedding are spread across many workers: partial progress, duplicate work after a retry, and what a checkpoint has to contain for a restart to be safe.