Tool Output Is Untrusted
A website, API or integration can return text that is false or malicious; tool data must not become higher-priority authority simply because a tool fetched it.
Frame the problem
Security starts with a concrete asset, attacker capability and trust crossing.
Why the system fails
Tool output is concatenated into instructions with no provenance, schema or downstream policy gate.
The important question is not “what is Tool Output Is Untrusted?” but “which assumption let untrusted data or an over-scoped identity cross external tool result → model context?” Trace the decision at the boundary, then constrain what can happen after the first control fails.
Design the control in layers
Start with the control closest to the interpretation or privilege boundary: Represent tool results as typed, provenance-labelled data Then add a control that reduces blast radius and telemetry that proves the decision was enforced.
The resulting design is not labelled secure. Record the identified controls, the known failure paths, the remaining exposure, and the evidence you would need during an incident.
| Prevent | Detect | Recover |
|---|---|---|
| Represent tool results as typed, provenance-labelled data · Validate schemas and constrain how fields influence actions · Re-authorize every downstream effect | Trace tool-result provenance into proposed actions | Contain the affected identity or component, scope impact from audit evidence, and preserve a regression test. |
Key points
- Asset: Agent policy, decision integrity and private context.
- Boundary: External tool result → model context
- Primary control: Represent tool results as typed, provenance-labelled data
- Detection signal: Trace tool-result provenance into proposed actions
- Always ask what limits damage when the primary control fails.
Boundary control exercise
This lesson uses the shared boundary-control exercise.
Follow the attack
Safe conceptual simulation: capability → missing control → crossed boundary → asset impact.
- 1Attacker starts with: An operator of an external site, API result or compromised integration.
- 2Tool output is concatenated into instructions with no provenance, schema or downstream policy gate.
- 3The weak or missing boundary control is crossed: External tool result → model context
- 4Impact: Indirect prompt injection, misinformation-driven actions or data exfiltration.
- Indirect prompt injection, misinformation-driven actions or data exfiltration.
Defend, detect, recover
One prevention is a single point of security failure. Layer it and make failure observable.
- • Represent tool results as typed, provenance-labelled data
- • Validate schemas and constrain how fields influence actions
- • Re-authorize every downstream effect
- • Trace tool-result provenance into proposed actions
- • Contain the affected identity or component.
- • Scope access from audit evidence.
- • Fix the boundary and add a regression test.
- • Misconfiguration and new access paths can bypass the intended control.
- • A privileged insider or compromised control plane may still reach the asset.