When Planning Helps
Explicit planning pays off on long-horizon tasks with dependencies or parallelisable subtasks; on simple tasks it only adds latency and cost — so decide with a rubric, evaluate plan quality, and enforce step budgets.
Where planning earns its cost
Planning is a bet that spending one call up front will save several calls (or several mistakes) later. The bet pays when the task has one of three properties.
- Long horizon — more than roughly five actions. Without a plan, the model drifts: it forgets the goal, repeats work, or stops early. An explicit plan re-injected each step (Dynamic Context Assembly) is the cheapest anti-drift device there is.
- Dependencies — step B needs the output of step A (find the tenant id → query usage → compute overage). A plan surfaces the order; direct execution guesses it and often calls B with a placeholder.
- Parallelisable subtasks — "compare pricing across five vendors" is five independent fetches. A plan lets the orchestrator run them concurrently; ReAct would do them one at a time, five round trips (Parallel vs Sequential Tool Calls).
Where it only adds latency and cost
For a one- or two-step task, a planner adds a full model round trip (typically 1–3 seconds and a few hundred output tokens) before any useful work starts, and produces a plan that says "1. look it up 2. answer". Users experience it as a slower product; your bill experiences it as a ~40–100% overhead on cheap tasks.
Worse, plans are made before evidence. On simple tasks the model would have found the right answer directly; forcing it through a plan gives it a chance to commit to a wrong decomposition. If the golden set (Golden Datasets) shows no accuracy gain from planning on a route, the correct move is to remove the planner from that route and let a Router Architecture send only complex requests to the planning path.
Evaluating plan quality
A plan is an intermediate artifact, and intermediate artifacts can be evaluated directly — cheaper and more diagnostic than only scoring final answers. Build a plan-level eval alongside the end-to-end one (Evaluating Agents: Testing Probabilistic Systems).
- Deterministic checks (Deterministic Evaluators): every step references an existing tool; arguments satisfy schemas; no step is destructive without an approval marker; step count ≤ budget; dependency graph is acyclic.
- Reference comparison: for golden tasks with a known good plan, measure step recall (did it include the necessary steps?) and step precision (how many were unnecessary?).
- Judge with a rubric (LLM-as-Judge): ordering correctness, missing preconditions, over-decomposition — scored 1–5 with the rubric in the prompt and calibrated against human labels.
- Outcome linkage: track which plans led to successful runs; a step type that correlates with failure is a planner prompt bug.
1type Step = { id: string; tool: string; args: Record<string, unknown>; dependsOn: string[] }2 3export function validatePlan(steps: Step[], tools: Record<string, { schema: (a: unknown) => boolean; destructive?: boolean }>, maxSteps = 10) {4 const errors: string[] = []5 if (steps.length > maxSteps) errors.push(`too many steps: ${steps.length} > ${maxSteps}`)6 const ids = new Set(steps.map((s) => s.id))7 for (const s of steps) {8 const t = tools[s.tool]9 if (!t) { errors.push(`unknown tool ${s.tool} in ${s.id}`); continue }10 if (!t.schema(s.args)) errors.push(`bad args for ${s.tool} in ${s.id}`)11 if (t.destructive) errors.push(`destructive step ${s.id} requires approval`)12 for (const d of s.dependsOn) if (!ids.has(d)) errors.push(`${s.id} depends on missing ${d}`)13 }14 if (hasCycle(steps)) errors.push('dependency cycle')15 return { ok: errors.length === 0, errors }16}Step budgets
Every planning strategy needs a hard budget, enforced by the orchestrator rather than requested from the model: maximum steps per task, maximum replans, maximum total tokens, and a wall-clock timeout (Budgets, Limits and Termination). When a budget trips, the agent returns what it has with an explicit "incomplete" status — never a fabricated success.
Budgets are also a design signal. If tasks routinely need 30 steps, the tool set is too fine-grained (combine get_user + get_plan + get_usage into get_account_summary) or the task should be split into a workflow with checkpoints. Set the budget from p95 step counts on successful runs plus a small margin, alert when it trips (Logging, Metrics and Alerts), and treat the alert as a bug report about the plan, not as a reason to raise the limit.
- Start at 8–10 steps and 2 replans; raise only with data from traces.
- Count tool calls and model calls separately — a "step" that fans out to 20 parallel fetches is not one step of cost.
- Return partial results with status; downstream code and users can act on "here is what I found so far".
Key points
- Plan when the task is long-horizon, has dependencies, or has parallelisable subtasks.
- On one- or two-step tasks a planner adds a round trip and cost with no accuracy gain — route around it.
- Evaluate plans directly: deterministic validation, step precision/recall against golden plans, rubric judges.
- Validate every plan before execution; the validator doubles as an eval.
- Enforce step, replan, token and time budgets in the orchestrator; return partial results with status.
- Persistently high step counts mean the tools or the task decomposition are wrong, not that the limit is too low.
When to use — and when not to
- Tasks with more than ~5 actions, or whose structure is unclear up front.
- Fan-out work (compare, gather, aggregate) that benefits from parallel steps.
- High-stakes tasks where a human should approve the plan before execution.
- Lookups, classification, single-tool actions — direct execution wins on every axis.
- Fixed sequences you already know — encode them as a workflow instead of asking a model to rediscover them.
- Latency-critical interactive paths where one extra round trip is unacceptable.
Failure modes
- Planner on every route: simple requests take 3 seconds longer and cost double.
- Plan executed without validation; an unknown tool name crashes step 4 after steps 1–3 mutated state.
- No step budget; an agent loops through replans until the timeout and returns nothing.
- Plan quality never measured; a prompt change silently makes plans 30% longer.
- Budget trips are handled by raising the budget instead of fixing the decomposition.