Infrastructure for Model and Agent Workloads

The first version of every agent product is a synchronous endpoint: the browser posts a message, the handler loops over model calls and tool calls, and eventually returns an answer. It works in a demo and fails in production for infrastructure reasons that have nothing to do with the model. The load balancer idle timeout cuts the connection at sixty seconds. The deploy that rolls the API pod kills nine in-flight runs mid-refund. The client retries a timed-out request and the agent runs the whole plan a second time.

The fix is the oldest pattern in the book, and it is the reason this lesson exists in an infrastructure domain rather than an AI one: accept the request, persist a run record, enqueue it, return 202 with a run id, and let a worker pool do the work. The agent runtime becomes a *worker* — a long-lived process that leases a job, executes the loop, checkpoints progress and reports terminal state. See Serverless and Database Connections for why the same workload is awkward under a function runtime with a hard execution ceiling, and the Architecture domain for the queue semantics themselves.

What is genuinely new compared with a normal background job is the fan-out from a single worker. One run touches a model provider over the public internet, a vector store, a blob store for attachments, and two internal APIs — each with its own identity, its own timeout, its own retry rule and its own bill. A worker that dies halfway leaves a partially executed plan, which is why every side-effecting tool call needs an idempotency key stored with the run, not generated per attempt.

The model credential and the tool execution environment are separate trust boundaries, and the most common production incident in this space comes from putting them in the same process. The worker needs the provider key to call the model. The sandbox needs to run whatever the model decided to run — a shell command, a generated SQL query, a fetch against a URL that came out of a retrieved document. If the sandbox inherits the worker's environment, then prompt-injected text in a support ticket has a path to the model key, the database credential and the instance metadata endpoint.

So the sandbox gets its own execution context with no ambient identity: a separate container or micro-VM, a network policy that denies egress except to an allow-listed proxy, a filesystem that is empty and non-persistent, and CPU/memory/wall-clock limits so a runaway loop costs a bounded amount. Credentials it genuinely needs arrive as short-lived, task-scoped tokens minted per run — not as an environment variable baked into the image. This is the infrastructure half of what the Security and Agentic domains teach as tool sandboxing; see Least Privilege in Infrastructure and Roles vs Static Keys for the identity mechanics.

Model credentials also behave unlike a database password in one important way: they are a *spending* credential. A leaked read-only API key costs you data. A leaked model key costs you data and an unbounded bill, charged to you, until you notice. Treat it as a first-class secret with rotation, per-environment scoping and a spend alert — the alert is the control that actually catches the leak.

Agent infrastructure moves the dominant cost line off compute. A conventional API bills you for instance-hours and a bit of egress. An agent platform bills you for tokens — a variable that scales with *conversation length and plan depth*, not with request count — plus retries, plus the runs that a bug turned into a loop. A single unbounded agent loop can spend more in an afternoon than the entire worker fleet spends in a month, and it will look like healthy traffic on every infrastructure dashboard.

That is why termination budgets belong in the infrastructure discussion rather than only in the prompt: a hard cap on steps, on tokens and on wall-clock per run, enforced by the runtime and not by the model's good intentions. Pair it with per-tenant quotas so one customer's pathological input cannot consume the shared model rate limit and starve everyone else — the model provider's rate limit is a shared, finite resource exactly like a connection pool.

Traces are the other non-negotiable. When a run produces a wrong refund, the question is *which step decided that*, and no metric answers it. Persist the full step sequence — prompts, tool calls, arguments, results, timings, token counts — to object storage keyed by run id, with a lifecycle rule that ages it into cheap storage. It is simultaneously your debugging tool, your cost attribution data and, when a regulator asks, your audit record. See Storage Lifecycle: Hot, Warm, Archive, Delete and Audit Trails.