Evaluation Data Pipelines
Production traces, sampled, privacy-filtered and versioned into an evaluation dataset. The privacy filter is the step most often skipped, and the version is what makes a score comparable across runs.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Where does an evaluation set come from, and what makes this month's score comparable with last month's?
The eval runner, which needs a fixed, versioned input set; the engineer deciding whether a prompt, model or chunker change is safe to ship; the reviewer who has to know whether a score movement is the system changing or the dataset changing; and the compliance function, for whom the evaluation store is a second copy of customer conversations living somewhere nobody classified (Evaluating Agents: Testing Probabilistic Systems).
One row is one evaluation case: an input, the context that should be retrieved, what a correct response looks like, and the provenance that says where it came from. One trace is not one case — a conversation can yield several cases or none — and conflating the two is why "we have five thousand evaluation cases" so often means "we have a copy of five thousand traces" (Grain: What Does One Row Represent?).
An engineer writes thirty questions and their expected answers in a spreadsheet, and the eval runner reads it. It takes an afternoon, it catches real regressions immediately, and it is the correct first version — building a sampling pipeline before you know which failures matter is how you get an elaborate harness measuring the wrong thing.
A business policy changes in June. The expected answers were written in March, so the suite now fails the cases where the system is right and passes the ones where it is wrong (Golden Datasets).
- A business policy changes in June. The expected answers were written in March, so the suite now fails the cases where the system is right and passes the ones where it is wrong (Golden Datasets).
- The expected answers were copied from what the system said at the time. The evaluation now measures whether the system still agrees with its own earlier output, which it does, forever.
- Cases were sampled uniformly from production. Successful runs dominate production, so the set contains almost none of the failures anyone cares about and the score is dominated by cases nothing was ever going to get wrong (Sampling Without Throwing Away the Evidence).
- Someone adds twenty cases before a release. The score moves and nobody can say whether the system improved or the denominator changed, because the dataset has no version (Semantic Changes).
- The set was built by copying traces into a spreadsheet in a shared drive. It contains customer names, order numbers and the contents of support conversations, and it is now the most widely readable copy of that data in the company (PII in Pipelines).
- A case ends up quoted in a prompt template as an example. The suite still passes it, and it stopped being evidence of anything the moment it entered the system it was measuring.
What is actually happening
- An evaluation set is a curated dataset built by a pipeline from captured data. Traces are captured rather than derived: they come from production events that will not happen again, so the sampler is the only source of new material and the trace archive's retention is the ceiling on what can ever be built (Agent Observability Data).
- Sampling is the design decision that defines what the metric means. Uniform sampling reproduces the production distribution, which is overwhelmingly successful runs, and produces a score that is high, stable and blind. Stratified sampling by outcome — every error, every low-confidence run, every negative feedback, a downsample of successes — produces a score that moves when something breaks (Sampling Without Throwing Away the Evidence).
- The privacy filter is a transformation stage in the pipeline, not a policy in a document. It has to run before the data lands in the evaluation store, because that store is read by more people, copied more often and retained far longer than the trace archive it came from (Data Minimization). Redaction, pseudonymisation and synthesis are three different transformations with different costs. Redaction removes and can destroy the case; pseudonymisation keeps structure and is reversible if the mapping leaks; synthesis rewrites the case around fabricated entities and can quietly change what is being tested (Data Masking, Tokenisation & Encryption).
- A score is a function of two versions: the system under test and the dataset. Comparing scores across dataset versions is comparing answers to two different sets of questions, and a chart that does not carry the dataset version is a chart that will be misread (Data Contracts).
- Case provenance is what makes the set maintainable: which trace it came from, which sampling rule selected it, who wrote the expected answer, when, and against which version of the business policy. Without provenance, an ageing case cannot be retired with confidence, so nobody retires anything and the suite decays (Data Lineage).
- Contamination is the failure with no technical symptom. A case whose expected answer was produced by the system under test measures self-consistency; a case that appears inside a prompt template measures memorisation. Both report excellent numbers (Golden Datasets).
From production traces to a scored run
The pipeline is short and every stage is a place where a score can quietly stop meaning anything. Read the guarantees column and note how little each stage promises: sampling promises a defined selection, the filter promises removal of what it detects, curation promises a human looked at it. None of them promises representativeness, and none of them can.
The privacy filter is placed where it is on purpose. It is not the last step before sharing, because sharing is not an event you control — the evaluation store is the copy, and it is duplicated into notebooks, result archives and repositories the moment it exists. The filter belongs before the first write, which is the only place it can be a control rather than an intention (Data Minimization).
The versioning stage is the one that makes the whole thing a measurement rather than an anecdote. Freezing an immutable snapshot with a changelog costs almost nothing and converts "the score went up" from an argument into a decomposition: which cases changed, under which dataset version, on which system version (Atomic Publish).
- 1Trace archive
Holds production runs: input, retrieved context, tool calls, output, outcome, feedback.
guarantees What was captured, for as long as retention allows. Captured data is not rebuildable, so the archive is the ceiling on everything downstream (Agent Observability Data).
fails by Head-based sampling or backpressure dropping, which discards uniformly and therefore discards the rare cases the evaluation set most needs (Sampling Without Throwing Away the Evidence).
- 2Sample
Selects candidate runs by an explicit rule — stratified by outcome, feedback, confidence and topic.
guarantees A defined selection whose rule is recorded on every case. Nothing about representativeness beyond what the rule was designed for.
fails by Defaulting to uniform, which fills the set with successes and produces a score that cannot move.
- 3Privacy filter
Redacts, pseudonymises or synthesises personal and confidential content across every field, not only the message body.
guarantees Removal of what it detects. This is a risk reduction, never an assurance of absence (Data Masking, Tokenisation & Encryption).
fails by Being written against the conversation while account numbers sit in tool arguments, retrieved chunks and error payloads (PII in Pipelines).
- 4Curate
Turns candidate runs into cases: the input, the context that should be retrieved, what a correct answer looks like, and the provenance.
guarantees That a human judged each expected answer, against a policy version recorded on the case.
fails by Copying the system's own output as the expected answer, which makes the case a self-consistency test forever.
- 5Version and freeze
Publishes an immutable dataset version with a changelog of what was added, retired and why.
guarantees Scores within one version are comparable. Across versions they are answers to different questions (Data Contracts).
fails by A living spreadsheet, so every score comparison mixes a system change with a dataset change and neither can be isolated.
- 6Run
Executes the system under test over every case and records per-case outcomes with both versions attached.
guarantees A reproducible result for a stated pair of versions, to the extent the system itself is deterministic (Deterministic Evaluators).
fails by Storing only an aggregate, which makes every future investigation a re-run rather than a query.
- 7Metrics store
Keeps results keyed on dataset version, system version and run, with per-case detail retained.
guarantees Decomposition: which cases moved, when, and under which change (Reconciliation).
fails by Plotting scores across dataset versions on one axis, which reports dataset edits as system improvements ("What Changed?" — Deploy Markers and the Invisible Deploys).
Seven stages and only one of them involves the model. This is a data pipeline with a curation step in the middle, and every failure listed above is a data-engineering failure with a familiar name.
One trace is not one case
The grain shifts three times between production and a score, and the shifts are where counting arguments come from. "We evaluate five thousand cases" and "we sampled five thousand traces" describe different things, and a metric defined over one and reported as the other is wrong in a way no test will catch (Grain: What Does One Row Represent?).
The table below is the same device this domain uses for fact tables. Each row is a legitimate unit, each supports a different question, and each produces a specific confidently wrong number when it is mistaken for its neighbour.
Note the last row in particular. The results table has its own grain — one row per case per system version per run — and getting it wrong is how a suite reports an average over a mixture of runs, or double-counts a case that was retried (Duplicate Rows).
| Stage | One row is | Breaks if |
|---|---|---|
| Trace | One complete agent run: input, retrieval, tool calls, output, outcome, feedback. | Multi-turn conversations are counted as runs. A single conversation with eight turns is one trace and eight opportunities for a wrong answer, and a metric over traces hides seven of them. |
| Turn | One user message and the system's response to it, inside a conversation. | Correctness depends on earlier turns. Extracting a turn as a standalone case loses the context that made the answer right, and the case then tests something the system was never asked (Context Construction & Grounding). |
| Sampled candidate | One trace or turn selected by a sampling rule, before any filtering or curation. | Candidates are counted as cases. Most candidates are discarded in curation, so the two numbers differ by a large and variable factor. |
| Evaluation case | One input, the context it should retrieve, a correct-answer definition, and provenance. | Near-duplicate cases enter the set. The score becomes weighted toward whatever topic was sampled most heavily, and nobody notices because the count went up (Deduplication). |
| Judgement | One assertion about one case: retrieval contained the answer, the response was faithful, the tool call was correct. | Several judgements per case are averaged into a per-case pass and then averaged again. The double aggregation hides which dimension regressed (Eval Metrics: What to Measure and How). |
| Run result | One case, one system version, one dataset version, one run, one outcome. | Retries are appended rather than keyed, so a flaky case appears several times and quietly gains weight in the aggregate. |
| Reported metric | One number for one run, over one dataset version. | It is compared with a number computed over a different dataset version — the single most common evaluation error, and it looks exactly like progress. |
Seven units in one pipeline. The rule that resolves nearly every dispute about evaluation numbers is to say which of these rows you are counting before saying how many there are.
The privacy filter, and what it will still let through
The filter is the step most often skipped, and the reason is structural rather than careless: it stands between an engineer and the debugging material they want, and skipping it has no immediate consequence. It is also the step with the longest tail of consequences, because the evaluation store outlives the trace archive and is read by far more people (Data Retention).
Every check below is real and every one has a blind spot that matters. Read the misses column as a specification for what a human sample review is for: detectors find categories they were written for, and the identifying material in a support conversation is frequently not in any category (Data Minimization).
The last row is the check on the check. A privacy audit run with the same detectors as the filter finds, by construction, exactly nothing — it is testing the implementation against itself. The audit has to use a different mechanism, and periodically a human reading a random sample, or it is a control that certifies its own output (Security-Safe Logging).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Structured detectors for the declared categories — contact details, payment identifiers, government numbers — across every field including tool arguments and retrieved chunks. | The declared categories of personal data do not survive into the store. | Emails, phone numbers, card and account identifiers pasted into messages, tool arguments carrying customer records, chunk text quoted back into the trace (PII in Pipelines). | Free-text identification — a person described unmistakably without any recognisable identifier — and internal reference numbers that identify a person to anyone with access to the ticket system. |
| Field coverage: every field present in the trace schema is either filtered or explicitly declared non-sensitive. | No field reaches the store unconsidered. | A new field added to traces by an unrelated change, which is the single most common way personal data starts flowing again after a filter was working (Schema Evolution). | A field correctly declared non-sensitive whose contents changed meaning later — the declaration is a point-in-time judgement with no expiry (Semantic Changes). |
| Contamination: no case input or expected answer appears verbatim in a prompt template, few-shot example or the retrieval corpus. | The cases are still held out from the system being measured. | Cases quoted into prompts as examples, evaluation questions added to the knowledge base, expected answers copied into documentation the agent retrieves (Golden Datasets). | Paraphrased leakage, which is the common form. A case reworded into a prompt is contamination that a text match cannot see. |
| Near-duplicate detection across cases within a dataset version. | The set is not silently weighted toward one topic. | A sampling run that pulled a burst of similar conversations, and repeated additions of the same case by different people (Deduplication). | Semantically identical cases with no surface similarity, which weight the score exactly the same way and cannot be found by matching. |
| Independent audit: different detectors from the filter, plus a human reading a random sample after every filter change. | The control actually worked on this data. | A filter that silently stopped matching after a schema change, a regex that never fired, a stage that was disabled during an incident and not re-enabled (Volume Anomalies). | Everything neither the second detector nor the reader recognises — which is why the sample is random and periodic rather than triggered, and why minimisation at capture time beats filtering at every stage after it. |
Four of the five checks are cheap and mechanical, and all four are blind to context. The residual risk is handled by collecting less in the first place: a field never captured cannot leak from a store nobody audited (Data Minimization).
Sampling decides what the number means
Every sampling rule produces a valid dataset and a different metric. The mistake is not choosing the wrong rule; it is choosing one implicitly and then reading the resulting score as though it answered a question it was never designed for.
The middle column is the one to fix in mind. "What the score then means" is a property of the sample, not of the metric definition, and two teams quoting the same headline pass rate from differently-sampled sets are reporting two unrelated facts (Two Dashboards, Two Numbers).
In practice the useful arrangement is two or three of these running side by side with different names and different jobs: a stratified core that gates releases, a rolling recent sample that detects drift, and an adversarial set that never gets easier. One number from one set cannot do all three (Regression Gates and Online Evaluation).
| Sampling rule | What the score then means | Good for | Blind spot |
|---|---|---|---|
| Uniform over production traces | An estimate of the production success rate, dominated by whatever is most common. | Reporting overall health to people who want one number. | Contains almost no failures, so it barely moves when something breaks — the least useful shape for a release gate (Sampling Without Throwing Away the Evidence). |
| Stratified by outcome: all errors, all negative feedback, downsampled successes | Performance on the cases that go wrong, deliberately over-weighted. | Regression gating and prioritising work. | Cannot be read as a production success rate, and anyone who reads it that way will conclude the system is far worse than it is. |
| Curated adversarial set, written by people who know the domain | How the system handles the hard cases someone thought of. | Finding failure classes before users do, and encoding known-hard behaviour permanently. | Reflects the imagination of its authors, ages badly against a changing product, and never gets easier because it was designed not to (Golden Datasets). |
| Rolling window of recent traces, rebuilt each period | Whether the system is keeping up with what people are currently asking. | Drift detection and catching a corpus that has stopped matching demand (Distribution Tests). | A moving yardstick: a score change may be a change in the questions rather than in the system, so it cannot gate a release. |
| Synthetic cases generated from the corpus | Coverage of content, independent of what anyone has asked yet. | Testing a new corpus before it has traffic, and covering documents no user has queried. | Measures the corpus against itself. Generated questions inherit the phrasing of the source text, which makes retrieval look easier than it is. |
| Everything, unsampled | The full production picture, at the cost of running every case on every change. | Small traffic volumes, where the whole population is affordable. | Cost grows with usage and with run frequency, so evaluation quietly becomes something you do less often — the opposite of the intent (Compute Waste). |
No row is correct in general and none of them is a compromise. Each answers a different question, which is why mature evaluation setups run several and never average them together.
Evaluation platforms and tracing products supply sampling, storage and judging, and the useful questions to ask of them are data-engineering ones: can the sampling rule be stratified by outcome, is the dataset versioned immutably, are per-case results retained across runs, does a privacy filter run before data lands in their store or after, and where does that store live. Feature coverage changes release by release — verify against current documentation rather than a comparison table.
How to build it
Most important first.
- Build the set from production traces through an explicit, scheduled pipeline rather than by hand, so it can be rebuilt, audited and refreshed — and so it drifts *with* reality instead of away from it (Evaluating Agents: Testing Probabilistic Systems).
- Sample by outcome, not uniformly. Keep every error, every run with negative feedback, every low-confidence answer and every escalation; downsample the successes deliberately and record the ratio, because the ratio is part of what the score means (Sampling Without Throwing Away the Evidence).
- Put the privacy filter immediately after sampling and before anything is written to the evaluation store, and test it as you would test any other transformation — with fixtures containing known personal data, asserting it does not survive (Data Tests, PII in Pipelines). Filter every field, not only the message body. Tool arguments, retrieved chunk text, error payloads and structured metadata are where the account numbers actually are, and a filter written against the conversation misses all of them (Data Classification).
- Version and freeze the dataset. A version is an immutable snapshot with a changelog saying what was added, what was retired and why; the eval runner records the dataset version beside every score (Atomic Publish). Keep two sets with different jobs: a stable core that changes rarely and gates releases, and a rolling recent set rebuilt from the last period of traces that detects drift. One dataset cannot be both a fixed yardstick and a fresh sample (Full Refresh vs Incremental).
- Record the policy version each expected answer was written against, so a policy change produces a list of cases to re-validate instead of a suite that silently encodes last year's rules (Slowly Changing Dimensions). Enforce holdout hygiene: evaluation cases must never appear in prompts, few-shot examples or retrieval corpora. Check it mechanically — the check is a text match against the corpus, and it finds things (Data Tests).
- Store results keyed on
(dataset_version, system_version, run_id)with per-case outcomes retained, so a score movement can be decomposed into which cases changed rather than argued about in the aggregate (Reconciliation).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The dataset guarantees only that its cases were true of the sample they came from at the moment they were sampled, against the policy in force then. Nothing about it is representative unless the sampling was designed to make it so (Golden Datasets).
- A score is comparable across runs only within one dataset version. Across versions it is a different measurement wearing the same axis label (Semantic Changes).
- The privacy filter guarantees the removal of what it detects, and detection is where every one of its failures lives. It is a reduction of risk, never an assurance of absence (Data Masking, Tokenisation & Encryption).
- The pipeline guarantees the set can be rebuilt only while the source traces still exist. Once retention expires, the cases in hand are all there will ever be from that period — evaluation sets inherit the non-rebuildability of captured data (Data Retention).
- Nothing guarantees the absence of contamination. A case whose expected answer came from the system, or that leaked into a prompt, produces a good score and no signal at all.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that must exist is a privacy audit of the evaluation store itself: run detectors for the categories your filter targets over the published cases, and have a human read a random sample every time the filter changes. The filter is the control; this is the check that the control worked (PII in Pipelines).
- It misses everything the detectors do not recognise — free-text identifiers, an internal case number that identifies a person to anyone with access to the ticket system, and the combination of harmless fields that is identifying together (Data Minimization).
- Pair it with a contamination check: no case input or expected answer appears verbatim in a prompt template, a few-shot example or the retrieval corpus. It is a text search and it regularly finds something (Golden Datasets).
- And a distribution comparison between the evaluation set and recent production traces — topic mix, input length, tool usage, outcome mix. When the two diverge, the score is measuring a population that has stopped existing (Distribution Tests).
- Evaluation sets have a freshness requirement like any dataset, and it points in two directions at once. The regression core must be stable, because a moving yardstick cannot detect a regression; the drift set must be recent, because a stale sample stops resembling production. Two sets, two policies (The Freshness SLO).
- The clock that matters for staleness is not when the set was built — it is when the world it encodes last changed. A case written against a policy that has since been revised is stale on the day the policy changed, however recently the file was touched (Semantic Changes).
- Trace retention bounds how fresh any rebuilt set can be and how far back a set can reach. An archive with short retention silently caps the diversity of every evaluation set you will ever build from it (Storage Lifecycle).
- Adding cases changes the denominator, so the aggregate score moves for reasons unrelated to the system. Publish per-case results and score movements decompose; publish only an average and every change is an argument (Two Dashboards, Two Numbers).
- Retiring cases is necessary and is the change people avoid, because retiring a case looks like removing evidence. Recording why each case was retired — policy superseded, duplicate, contaminated, invalid — is what makes it defensible (Dataset Documentation).
- Changing a metric definition or a judging rubric is a semantic change with no schema footprint. Everything before and after is incomparable, and the time series needs a marker at the change ("What Changed?" — Deploy Markers and the Invisible Deploys).
- The case schema itself evolves gently: extra fields are additive and old cases simply lack them. What is not gentle is changing the meaning of an existing field — "expected answer" becoming "acceptable answer" changes every case at once (Semantic Changes).
- The set is rebuildable from traces if — and only if — the sampling rule, the filter version and the source traces still exist. Record the rule and the version on every case, or a rebuild produces a different dataset that shares a name with the old one (Reprocessing vs Retrying).
- The labels are not rebuildable. Human judgements about what a correct answer looks like are the expensive part and they should be treated as a first-class asset, backed up independently of the pipeline that assembled them (Backup Strategy).
- A privacy incident in the evaluation store is recovered by deleting affected cases everywhere they were copied — repositories, notebooks, result archives — which is only tractable if the store is one place with an access list rather than a folder people copy from (Deletion Requests).
What can go wrong
- Uniform sampling, producing a set with almost no failures in it and a score that cannot move (Sampling Without Throwing Away the Evidence).
- A privacy filter applied to message bodies only, while tool arguments and retrieved chunk text pass through untouched.
- Expected answers written from the system's own output, making the suite a self-consistency test.
- An unversioned dataset, so every score comparison silently mixes system change with dataset change.
- Cases quoted into a prompt template as examples, which converts them from evidence into memorisation (Golden Datasets).
- The mitigation failing: a privacy audit run with the same detectors as the filter, which by construction finds exactly nothing it did not already remove.
- "The evaluation set is a test fixture." It is a dataset with a grain, a schema, a provenance, a freshness requirement, a privacy classification and a version. Treating it as fixtures in a repository is how it ends up unversioned and full of personal data (Data Products).
- "We sampled from production, so it is representative." Uniform sampling from a production distribution dominated by successes produces a set that is representative of nothing anyone is trying to measure (Sampling Without Throwing Away the Evidence).
- "The score went up, so the system improved." Only if the dataset version was identical. Cases added, cases retired and a rubric change all move the number (Semantic Changes).
- "We will add the privacy filter before we share it more widely." The store is the copy, and the copies happen before the intention. The filter belongs before the write, which is the only place it can be enforced rather than remembered (PII in Pipelines).
- "We can build the evaluation set later." Cases come from traces, traces are captured rather than derived, and traces from before you started collecting them do not exist. Later is a different and worse dataset (Agent Observability Data).
- The evaluation store is a curated, long-lived, widely-read copy of production conversations. Unless it is classified, access-controlled and retained deliberately, it is the least governed and most readable copy of sensitive data in the platform (Data Classification, Data Access Control).
- Deletion requests must reach evaluation cases derived from a subject's traces. That is only tractable when each case records the trace it came from, which makes provenance a governance requirement rather than an engineering nicety (Deletion Requests, Data Lineage).
- A privacy filter is a control that needs evidence it works: a tested transformation, a change log, and an independent audit of its output. A filter nobody audits is an assumption with a function name (Security-Safe Logging).
Operating it
- Dataset version recorded beside every published score, and a chart that refuses to plot two versions on one line ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Per-case results retained per run, so a score movement decomposes into which cases flipped rather than into an argument about an average (Two Dashboards, Two Numbers).
- Age distribution of cases and the count whose recorded policy version has since been superseded — the direct measure of suite decay (Freshness Monitoring).
- Sampling ratios actually achieved per stratum, compared against the intended ones. A stratified sampler that silently degrades to uniform is a common and quiet failure (Distribution Tests).
- Privacy filter hit rates per field and per category, because a filter whose hit rate drops to zero after a schema change has stopped seeing the field it was written for (Volume Anomalies).
- Divergence between the evaluation set's distribution and recent production traffic, reviewed on a schedule rather than when someone is suspicious (The Data Quality Dashboard).
- At ten times the case count, run cost begins to gate how often evaluation happens, which is the wrong thing to economise on. Tiering — a fast core on every change, the full suite nightly — keeps the gate cheap and the coverage broad (Cost vs Freshness).
- At ten times the trace volume, sampling stops being a scan and becomes an incremental job with a high-water mark over the archive (Incremental Processing, The High-Water Mark).
- As the number of things being evaluated grows — several prompts, several models, several corpora — the result store becomes the interesting dataset, and it needs a grain statement of its own: one row per case per system version per run (Grain: What Does One Row Represent?).
- Privacy risk scales with copies, not with size. A set of five hundred filtered cases in ten notebooks is a bigger exposure than fifty thousand cases in one access-controlled store (Data Minimization).
- Three drivers, and they are ordinary ones. Bytes scanned by the sampler over the trace archive, human labelling time for expected answers, and per-item external compute for every eval run over every case (What Actually Drives Data Platform Cost).
- Every case added is paid on every subsequent run, forever, by every change that triggers an evaluation. A large suite is a recurring cost and the marginal value of the thousandth near-duplicate case is close to zero (Compute Waste).
- Model-judged metrics multiply the per-run cost by the number of judgements per case, which is why the deterministic checks that can be expressed as assertions should be, and only the genuinely subjective parts should reach a judge (Deterministic Evaluators).
- Retaining per-case results for every run costs storage that grows with runs times cases, and it is what makes decomposition possible. Keep it, and put a lifecycle policy on it (Storage Lifecycle).
- Stratified sampling produces a score that moves when something breaks and a score that cannot be read as a production success rate. Those are different measurements and trying to get both from one number is how evaluation dashboards become misleading.
- Aggressive privacy filtering protects the store and can destroy the case — a support conversation with every entity removed may no longer test anything. Synthesis preserves the shape and risks quietly changing what is being tested.
- A frozen core set is a reliable yardstick that slowly stops resembling production. A continuously refreshed set tracks reality and cannot detect a regression. Running both costs twice and is the only arrangement that answers both questions.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe pipeline shape — sample, filter, curate, version, run, store per-case results — holds for any agent system regardless of model, framework or metric. What differs is what a case contains and how correctness is judged, neither of which changes the requirement that the dataset be versioned and the filter be a stage rather than a policy.
- ORG-SPECIFICWhat must be removed before traces may be used for evaluation is decided by classification and regulatory context rather than by engineering: a consumer support product and an internal developer tool have completely different filters, and the same pipeline serves both. What is not org-specific is that the filter runs before the write.
- SCALE-SPECIFICFor a single team with a handful of daily conversations, a curated spreadsheet reviewed monthly is a better use of effort than a sampling pipeline, because a human can read the whole traffic. The pipeline becomes necessary once nobody can read production any more, which is a traffic threshold rather than a data-volume one.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the release gate this dataset feeds: what a failing evaluation should block, how a suite is run in a delivery pipeline, and why a gate whose input is a living spreadsheet cannot be trusted to block anything.
- — Distributed Systems owns why the trace stream this pipeline samples from is at-least-once and unordered, and what that means for a sampler that is trying to select a representative set from it.