Router Architecture
Classify the intent once with a cheap model, then dispatch to a specialised handler — plain code, a single LLM call, a workflow, or an agent — so that each request pays only for the machinery it needs.
The shape
A router is a two-stage system. Stage one classifies the request into one of a small, fixed set of intents. Stage two dispatches to the handler registered for that intent. The handlers are heterogeneous on purpose: order_status might be a database lookup with no LLM at all, draft_reply a single model call, refund a workflow with an approval gate, and open_ended_research a bounded agent.
The router itself should be the cheapest thing that classifies accurately: a small model with structured output, a fine-tuned classifier, or even keyword rules for unambiguous cases. Its output is a label plus a confidence, not prose. Everything expensive is behind the dispatch.
This is the architecture that most directly implements the escalation ladder in Choosing the Right Abstraction: each intent gets the lowest rung that solves it, and the router is what lets different rungs coexist in one product.
Designing the intent set
Keep the label set small — 5 to 15 intents — and mutually exclusive. Classification accuracy drops sharply when labels overlap ("billing question" vs "invoice question"), and the failure is invisible: the request lands in a handler that almost fits and produces a plausible wrong answer. Write a one-line definition and three examples per intent; that text becomes the router prompt and the labelling guide for evaluation data.
Always include an other / unclear intent and route it to a clarification turn or a human, never to the most general handler. Combine the label with a confidence score and a threshold; below the threshold, ask a clarifying question. This single guard removes most of the embarrassing failures of routed systems.
Measure the router separately from the handlers. A 500-example labelled set, a confusion matrix, and per-intent precision/recall is a small afternoon of work and it is the only way to know whether a bad answer came from routing or from handling, as Eval Metrics: What to Measure and How lays out.
- 5–15 intents, mutually exclusive, each with a definition and examples.
- Structured output:
{ intent: enum, confidence: number, entities?: {...} }— never free text. unclearis a first-class label routed to clarification, not to a catch-all agent.- Extract entities (order id, date range) in the same call so handlers start with what they need.
Cheap model for routing
Routing is a short-input, short-output classification, which is exactly what small models are good at. A small model at a few hundred input tokens and ~20 output tokens costs a fraction of a cent and returns in 200–500 ms; the same call with a frontier model is 10–30× the price and slower, with no accuracy gain on a well-defined label set. Route cheap, spend on the handler.
When accuracy on a specific label matters (for example anything touching money), add a second cheap check: a rule that verifies an order id is present before dispatching to refund, or a second-opinion classification when confidence is between 0.5 and 0.8. Cascades like this are covered in Fallbacks, Caching and Model Routing.
Cache routing decisions for repeated inputs by normalised text hash. Support traffic is highly repetitive; a small LRU in front of the router removes a measurable share of calls.
1type Intent = 'order_status' | 'draft_reply' | 'refund' | 'research' | 'unclear';2type Route = { intent: Intent; confidence: number; entities: Record<string, string> };3 4const handlers: Record<Intent, (req: string, r: Route) => Promise<string>> = {5 order_status: async (_, r) => formatStatus(await db.orders.find(r.entities.orderId)), // plain code6 draft_reply: async (req) => llm({ system: REPLY_SYSTEM, user: req }), // one call7 refund: async (req, r) => runWorkflow(refundGraph, { req, orderId: r.entities.orderId }), // workflow + gate8 research: async (req) => runAgent(req, { tools: researchTools, maxSteps: 10 }), // bounded agent9 unclear: async () => 'Could you tell me a bit more about what you need?',10};11 12export async function handle(req: string): Promise<string> {13 const route = await smallModel.structured<Route>({ system: ROUTER_PROMPT, user: req, schema: RouteSchema });14 const intent = route.confidence >= 0.75 ? route.intent : 'unclear';15 span('route', { intent, confidence: route.confidence });16 return handlers[intent](req, route);17}Properties
Complexity is low to moderate: one classifier, a registry, and whatever each handler needs. The handlers can be developed, tested, and deployed independently, which is a real organisational advantage. Latency is one cheap call plus the handler, so for code-backed intents the whole request is well under a second, and the p95 is set by the slowest handler, not by the architecture.
Cost is the lowest of any architecture that still supports open-ended requests, because most traffic never touches an expensive model. Reliability is high for well-separated intents and degrades exactly with routing accuracy. Debuggability is very good: the trace has a routing span with the label and confidence, then the handler's own span; misroutes are the first thing you check and they are visible.
The limit is that intents must be enumerable. A router cannot handle a request that spans two intents ("check my order and draft a complaint") without either a multi intent that decomposes it in code or escalation to a Supervisor Architecture — which is precisely the discriminating question between the two.
- Complexity: 2 — classifier plus a handler registry.
- Latency: 1–2 — one small call plus the chosen handler.
- Cost: 1–2 — most requests hit cheap handlers.
- Reliability: 4 — bounded by routing accuracy; measurable.
- Debuggability: 5 — misroutes are visible in one span.
Key points
- Classify once with a cheap model, dispatch to a handler; handlers can be code, a single call, a workflow, or a bounded agent.
- Keep intents few, mutually exclusive, and documented with examples; include
unclearand route it to clarification. - Emit structured output with a confidence score and threshold it; entities extracted in the same call feed the handler.
- Measure routing accuracy separately from handler quality with a labelled set and confusion matrix.
- Lowest cost architecture that still handles open-ended traffic; latency is set by the handler, not the router.
- Fails on multi-intent requests and on overlapping labels — that is when a supervisor or a decomposition step is justified.
When to use — and when not to
- Traffic falls into a small number of recognisable intents with very different handling costs.
- Some intents are best served by deterministic code and must not pay for an LLM.
- Different intents need different risk controls (an approval gate for refunds, none for status checks).
- You want to add capabilities incrementally by registering new handlers.
- Requests routinely combine several intents that must be solved together.
- The intent space is open-ended or shifts weekly; the label set will never be stable.
- One handler serves 95% of traffic — the router is overhead; call that handler directly and special-case the rest.
- Handlers need to share a running conversation state that the router discards on every turn.
Failure modes
- Overlapping intents produce silent misroutes that look like handler bugs.
- No confidence threshold: low-confidence guesses are dispatched and answer the wrong question fluently.
- Catch-all agent as the fallback absorbs every ambiguous request and becomes the most expensive path.
- Router prompt drifts from the labelled eval set; accuracy decays unnoticed after a prompt edit.
- Entities extracted by the router are wrong (order id typo) and the code handler fails with a cryptic error.
Tradeoffs
Ratings assume routing accuracy is measured; an unmeasured router is reliability 2.