Is version B actually better?
“How would you determine whether version B of an agent is actually better than version A?”
What this tests
- Understanding that a single demo run is not evidence for probabilistic systems
- Golden datasets, metrics, repeated runs, and statistical thresholds
- Multi-dimensional comparison: quality, cost, latency, safety
- Awareness of judge bias, dataset drift, and online validation
Answers by level
Read the beginner answer first and notice what is missing.
Because outputs vary run to run, I need a golden dataset (representative inputs with expected outcomes or rubrics, including hard cases and past failures), metrics that fit the task (exact match or deterministic checks where possible, LLM-as-judge with a rubric where not, plus tool-selection accuracy and task completion), and enough runs to separate signal from noise: several runs per input, reporting mean and variance, not a single pass. See Evaluating Agents: Testing Probabilistic Systems, Golden Datasets, and Eval Metrics: What to Measure and How.
Then compare on more than quality: cost per successful task, p50/p95 latency, steps per task, error and cap-hit rates, and safety evals (injection resistance, destructive-call rate). B that is 3% more accurate and twice as slow may not be better for the product. Regression tests catch cases A handled that B breaks; the aggregate can rise while specific important cases fail. See Regression Gates and Online Evaluation.
Finally, validate online: a shadow or A/B rollout with the same metrics plus user signals (thumbs, escalation rate, task abandonment), because the golden set never fully matches production traffic.
Green flags · Red flags
- Golden dataset with hard cases and past failures
- Multiple runs per input, variance reported, statistical threshold
- Guardrail metrics: cost, latency, safety, cap-hit rate
- Slices by category and regression checks on specific cases
- Calibrated judge, deterministic evaluators preferred
- Online validation via shadow or A/B
- Compares a handful of single runs
- Uses an uncalibrated LLM judge as the sole arbiter
- Only looks at accuracy
- No held-out data or awareness of tuning on the test set