HITLhitlconfidencecalibrationthresholdsrouting

Confidence Thresholds

Model self-reported confidence is poorly calibrated; route to humans using external signals — evaluator scores, retrieval similarity, validation results — with thresholds set on eval data, not by feel.

Interview question
Progress

Why "confidence: 0.92" means little

Asking the model to output a confidence number is cheap and tempting. It is also mostly noise: LLMs are trained to sound assured, they report high confidence for fluent wrong answers, and the number drifts with prompt wording and model version. Calibration studies routinely find self-reported 90% confidence corresponds to far lower accuracy on hard items and the mapping changes between models.

Token log-probabilities are somewhat better for short classification outputs, but they measure "how predictable is the next token", not "is this action correct". For a tool call with four arguments there is no single probability that means what you want.

Signals that actually carry information

Use signals produced outside the generating model, each tied to a specific failure mode.

  • Retrieval scores: top-1 cosine similarity or reranker score below a floor means the answer is probably ungrounded → route to human or say "I don't know" (Reranking, Context Construction & Grounding).
  • Validation results: schema validation, argument range checks, dry-run output ("matches 1,204 records") — hard facts about the proposed action (Argument Validation).
  • Evaluator signals: a separate judge model or deterministic checker scoring the output for groundedness, policy compliance or citation validity (LLM-as-Judge, Deterministic Evaluators).
  • Agreement: sample the answer 3 times at temperature; disagreement is a strong uncertainty signal for classification-style tasks.
  • Novelty: input far from anything in the golden set (embedding distance) — the agent is off its evaluated distribution.

Setting thresholds from evals

A threshold converts a signal into a routing decision: auto-execute above, human review below. Choose it on a labelled eval set by looking at the trade-off between automation rate and error rate among auto-executed items. Plot accuracy as a function of the signal; pick the point where auto-executed accuracy meets the risk class's requirement (say 99% for class 2), and read off the automation rate you get for free.

Then monitor the signal distribution in production. If the fraction below threshold doubles, either the traffic changed or the retriever/model degraded — either way a human should look before the threshold is moved (Regression Gates and Online Evaluation).

Choose the threshold that meets a target precision on eval data
1def pick_threshold(scores: list[float], correct: list[bool], target_precision: float) -> float:
2    """Lowest score s such that items with score >= s are correct at >= target_precision."""
3    pairs = sorted(zip(scores, correct), reverse=True)
4    best, hits = 1.01, 0
5    for i, (s, ok) in enumerate(pairs, start=1):
6        hits += ok
7        if hits / i >= target_precision:
8            best = s                      # widen automation while precision holds
9    return best
10 
11# route at runtime
12def route(signal: float, threshold: float) -> str:
13    return "auto" if signal >= threshold else "human"

Combining signals

Different signals catch different failures, so combine them with any-fail logic rather than an average: low retrieval score OR failed validation OR judge below floor → human. Averaging lets a confident judge mask a missing retrieval. Keep the rule readable; a two-line policy that ops can reason about beats a learned combiner nobody can explain in an incident review.

Key points

  • Self-reported confidence is uncalibrated and drifts with prompts and models; do not route on it.
  • Use external signals: retrieval scores, validation results, evaluator scores, sampling agreement, novelty.
  • Set thresholds on labelled eval data to hit a target precision per risk class.
  • Combine signals with any-fail logic, not averages.
  • Monitor the signal distribution; a shift is an alert, not a reason to move the threshold.

When to use — and when not to

Use it when
  • Deciding which outputs or actions can skip human review.
  • RAG answers where groundedness varies by query.
  • Classification or extraction tasks with a measurable accuracy requirement.
Avoid it when
  • Class 3 actions — no signal makes an irreversible large action safe to auto-run.
  • Tasks with no labelled eval set yet; you cannot calibrate what you have not measured.
  • When the signal is produced by the same model turn that produced the answer.

Failure modes

  • Routing on a self-reported confidence field that is always 0.9.
  • Threshold tuned once, never revisited after a model upgrade.
  • Averaged signals hide a failed validation.
  • Judge model shares the generator's blind spots (llm-judge-pitfalls is the interview question).
  • Automation rate optimised without watching the error rate of auto-executed items.
← In-the-Loop vs On-the-Loop and Escalation