HITLhitlhotlescalationslafallback

In-the-Loop vs On-the-Loop and Escalation

Human-in-the-loop blocks on approval; human-on-the-loop monitors and intervenes; both need escalation paths, SLAs and a defined fallback when nobody answers.

Interview question
Progress

Two oversight modes

In-the-loop (HITL): the agent stops and waits; nothing happens without a human decision. Maximum safety, maximum latency, and it needs a human to be available. Appropriate for class 2–3 actions from Approval Gates and Risk Classes.

On-the-loop (HOTL): the agent acts autonomously; humans watch dashboards, sample traces, and can pause, roll back or take over. Throughput is preserved; safety relies on detection speed and reversibility. Appropriate for class 0–1 actions, and for class 2 once evals and reversal rates justify it.

Most systems mix both by class, and shift actions from HITL to HOTL over time as evidence accumulates — the reverse shift after an incident should be a one-line config change, not a redeploy.

  • HITL question: "May I do this?" — answered before execution.
  • HOTL question: "Is this going well?" — answered by monitoring after execution (Logging, Metrics and Alerts).
  • HOTL is only safe when the action is reversible within the detection window.

Escalation paths

An escalation is a routed request for a human with defined recipients, order and timers. First-line: the user who started the run (they have context and are usually online). Second-line: a team queue (support, ops, finance) with a rota. Third-line: an on-call owner for class 3 actions. Each hop has an SLA — for example 5 minutes, 30 minutes, 2 hours — after which the request moves on.

Escalations must carry enough context to decide without re-reading the transcript: the preview, the risk class, why the agent thinks the action is needed, and what happens if nobody approves. Route by class and domain, not by "whoever is free".

Escalation with SLAs and fallback
yesnoyesnoyesnoApproval requestedRequesting userAnswered within 5 min?Team queueAnswered within 30 min?On-call ownerAnswered within 2 h?Execute as decidedFallback: safe default
UserLLMAgentToolDataDecisionHumanGuardrail

When no human is available

Define the fallback per risk class before launch. The safe default is almost always "do not execute; park the run; notify". For some actions a time-boxed alternative exists: hold the refund and email the customer that it is being reviewed; save the post as a draft; open a ticket instead of deploying. "Auto-approve after timeout" is only acceptable for class 1 and must be explicitly configured, never the default.

Parked runs need a durable store, a resume path and an expiry. A run that resumes three days later may be operating on stale data; re-run the preview and require fresh approval if it changed (Budgets, Limits and Termination).

Per-class escalation policy with an explicit fallback
1type Policy = { chain: { queue: string; slaMs: number }[]; onTimeout: 'reject' | 'park' | 'auto-approve' }
2 
3const POLICY: Record<number, Policy> = {
4  1: { chain: [{ queue: 'requester', slaMs: 5 * 60_000 }], onTimeout: 'auto-approve' },
5  2: { chain: [{ queue: 'requester', slaMs: 5 * 60_000 }, { queue: 'support', slaMs: 30 * 60_000 }], onTimeout: 'park' },
6  3: { chain: [{ queue: 'requester', slaMs: 5 * 60_000 }, { queue: 'finance', slaMs: 30 * 60_000 }, { queue: 'oncall', slaMs: 2 * 3_600_000 }], onTimeout: 'reject' },
7}

Key points

  • HITL blocks before execution; HOTL monitors after execution and needs reversibility.
  • Assign modes per risk class and shift them based on evidence.
  • Escalation = ordered recipients + SLAs + context to decide.
  • Define the no-human fallback per class before launch; default is do-not-execute.
  • Parked runs are durable, resumable, and re-validated on resume.

When to use — and when not to

Use it when
  • Any gated agent that runs outside business hours or across time zones.
  • Moving an action from approval to monitoring after evals justify it.
  • Post-incident hardening: adding a hop or shortening an SLA.
Avoid it when
  • Do not use HOTL for irreversible actions — detection after the fact does not help.
  • Do not escalate class 1 actions to on-call; that burns the rota's attention.
  • Do not set auto-approve on timeout for anything above class 1.

Failure modes

  • Approval requests to an empty queue; runs silently pile up.
  • Auto-approve-on-timeout applied globally "to keep things moving".
  • Escalation without context; approver re-reads a 20k-token transcript.
  • HOTL dashboards nobody looks at; alerts without owners.
  • Resumed run executes on stale preview.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy
← Approval Gates and Risk ClassesConfidence Thresholds →