Evalsevalsmetricslatencycostsafety

Eval Metrics: What to Measure and How

A catalogue of agent metrics — outcome, trajectory, retrieval, efficiency and safety — with a precise definition and computation for each, so a number means the same thing across versions.

Interview question
Progress

Metrics need definitions, not names

"Accuracy went from 81% to 85%" is meaningless unless everyone agrees what counts as correct, on which dataset, scored by what. Each metric below has a definition (what it measures), a computation (how a single case is scored and how cases aggregate) and a note on who can score it — code, a judge, or a human.

Pick metrics in layers: an outcome metric that captures what the user wanted, a few trajectory metrics that explain how the agent got there, efficiency metrics that bound cost and latency, and safety metrics that must stay at zero. A change that improves the outcome while doubling steps is not obviously a win.

Outcome and trajectory metrics

Outcome metrics score the end result. Trajectory metrics score the path — which tools were called, with what arguments, in how many steps. Trajectory metrics are what make agent evals different from plain LLM evals: two runs can reach the same answer, one in 3 steps with the right tools and one in 14 steps with a lucky guess.

  • Task success: did the run achieve the goal? Per case: 1/0 by a deterministic outcome check (record exists, ticket closed, test passes) or a judge with a rubric. Aggregate: mean → success rate. The headline metric for end-to-end evals.
  • Answer correctness: is the final text right? Per case: exact/normalised match for closed answers, judge-vs-reference score (0–1) for open answers. Aggregate: mean. Distinct from success — an agent can answer correctly but fail to perform the required action.
  • Tool selection accuracy: fraction of steps where the tool chosen matches the expected tool for that state. Per case: matches / expected tool calls. Aggregate: micro-average across all steps. Catches a router or description bug directly (wrong-tool-selected).
  • Tool argument correctness: for each correctly selected tool, do the arguments validate against the schema and match the expected values (exact, or within tolerance for numbers/dates)? Per case: correct args / tool calls. Detects tool-argument-drift after prompt or model changes.
  • Number of steps: LLM calls (or loop iterations) per run. Report median and p95, not mean — one runaway run dominates a mean. A rising p95 with flat success means the agent is getting lucky, not better.
  • Hallucination rate: fraction of claims (or answers) not supported by the provided context or tool results. Per case: judge extracts claims and checks each against the context; unsupported / total. Requires the trace to include the context the model saw.

Retrieval, efficiency and safety metrics

Retrieval metrics score the RAG layer in isolation (see RAG Evaluation). Efficiency metrics should be recorded on every run, eval or production, because they are free once traces exist. Safety metrics are usually thresholds, not averages: the acceptable number of exfiltrated secrets is zero.

  • Retrieval quality: recall@k (gold chunk in top k), precision@k, MRR (1 / rank of first relevant chunk). Needs a labelled query→relevant-chunk set. Compute per query, average over queries.
  • Latency p50 / p95: wall-clock from request to final answer. p50 is the typical experience; p95 is what your slowest 1 in 20 users get and what timeouts must be set against. Never report a mean latency.
  • Tokens: prompt + completion tokens per run, split by call. Track prompt tokens per step — growth over steps is the signature of context bloat (context-overflow-degradation).
  • Cost: tokens × price per model, plus tool costs (search API, sandbox minutes). Report cost per task, median and p95, and total per eval run. A 4% accuracy gain that triples cost per task needs a business case.
  • Safety violations: count of runs that called a forbidden tool, leaked a secret or PII pattern, executed a blocked action, or produced output failing the output guardrail. Aggregate as a count with a hard threshold of 0 for the severe categories.

Computing metrics from traces

Every metric above is a function of the trace plus the dataset entry. That is the argument for storing traces in a structured form: metrics become pure functions you can re-run, version and unit-test.

Trajectory metrics from a trace; aggregation happens outside.
1def trajectory_metrics(trace, case):
2    calls = [s for s in trace.spans if s.kind == "tool"]
3    expected = case.expected_tools            # ordered list of (name, args)
4    selected = sum(1 for s, (name, _) in zip(calls, expected) if s.name == name)
5    args_ok = sum(1 for s, (name, args) in zip(calls, expected)
6                  if s.name == name and s.input == args)
7    return {
8        "tool_selection": selected / max(len(expected), 1),
9        "tool_args": args_ok / max(selected, 1),
10        "steps": len([s for s in trace.spans if s.kind == "llm"]),
11        "tokens": sum(s.tokens_in + s.tokens_out for s in trace.spans if s.kind == "llm"),
12        "cost_usd": sum(s.cost_usd for s in trace.spans),
13        "latency_ms": trace.end_ms - trace.start_ms,
14    }

Key points

  • Each metric needs a written definition, a per-case computation and an aggregation rule.
  • Layer metrics: outcome (success, correctness), trajectory (tool selection, arguments, steps), efficiency (latency, tokens, cost), safety (violations).
  • Use medians and p95 for steps, latency and cost; means hide runaway runs.
  • Tool selection and argument correctness are the metrics that localise agent regressions.
  • Hallucination rate requires the trace to capture the exact context the model saw.
  • Safety metrics are thresholds, not averages.

When to use — and when not to

Use it when
  • Defining the scorecard before building an eval suite.
  • Deciding whether a model upgrade is a net improvement across quality, cost and latency.
  • Setting SLOs and alert thresholds for production (Logging, Metrics and Alerts).
Avoid it when
  • Do not report a single blended score; it hides which layer regressed.
  • Do not compute retrieval metrics on the final answer — they need labelled chunks.
  • Do not use mean latency or mean cost for anything user-facing.

Failure modes

  • Success rate rises while p95 steps doubles — the agent is brute-forcing, and cost follows.
  • Answer correctness scored by a judge without a reference, so the judge rewards confident wrong answers.
  • Tool argument correctness computed with exact string match on free-text arguments, producing false failures.
  • Cost tracked in tokens only, ignoring tool API charges that dominate the bill.
  • Safety violations averaged into a percentage and treated as "acceptable at 0.5%".
← Evaluating Agents: Testing Probabilistic SystemsGolden Datasets →