The question this answers
Should this workload call a hosted model API, deploy to managed inference, self-host on our own compute, or run a dedicated accelerator cluster?
Product needs a summarization and classification feature in the application. Legal needs to know whether customer text leaves the company's boundary. Finance needs to know whether the cost is per request or per month. Nobody has asked for a GPU cluster; someone has already started pricing one.
A decision framed on five axes the business actually cares about — cost shape, latency, control, data boundary and scaling behaviour — instead of on which option sounds most serious.
Four categories, and what each one really is
Strip the product names away and there are four ways to run a model. A hosted model API is someone else's model behind someone else's endpoint: you send text, you get text, you pay per token, you own nothing. Managed inference is your model artifact deployed onto a provider's serving platform: they own the hardware, the drivers, the scaling and the endpoint; you own the model and the configuration. Self-hosted inference is your model in your container on your compute — usually accelerator instances you rent — where you own everything above the hypervisor. A dedicated accelerator cluster is that, at scale, with capacity commitments, a scheduler and a team.
The categories differ mainly in where the Shared Responsibility line falls, and the boundary moves in exactly the way it does for databases: the further right you go, the more you own and the more you can tune. What is unusual about this particular decision is that the leftmost option is not just the simplest — for most workloads it is also the cheapest, because a hosted API charges nothing when idle and a rented accelerator charges everything.
The dishonest version of this comparison presents self-hosting as the mature choice and the API as a starter option. That is backwards for most teams. The API is the correct default; the other three exist for specific, nameable reasons — the data may not leave the boundary, the volume has crossed the point where per-token pricing loses to a saturated device, or the model is one nobody hosts. If you cannot state which of those applies, the answer is the API.
| Hosted model API | Managed inference | Self-hosted inference | Dedicated cluster | |
|---|---|---|---|---|
| Cost shape | Pure usage — zero at idle | Mostly fixed per endpoint-hour | Fixed device-hours, idle or not | Committed capacity, large fixed floor |
| Cheapest when | Volume is low, spiky or unknown | Volume is steady and moderate | Volume is steady and high | Utilization is consistently high |
| Latency | Public internet round trip + provider queue | In-region, warm endpoint | In-VPC, tunable, warm | In-VPC, lowest and most predictable |
| Control over the model | None — versions change under you | Full over the artifact, none over hardware | Full | Full, including topology |
| Does data leave your boundary? | Yes — to the provider | To your provider, inside your account | No | No |
| Scaling behaviour | Provider's, capped by rate limit | Configurable, cold start on scale-out | Yours to build, minutes to add a node | Yours, bounded by committed capacity |
| Ops burden | Almost none | Low — endpoint config and versions | High — drivers, batching, capacity | Very high — a platform team |
| Team size it suits | Any | Small to medium | Medium with a platform function | Large, or a core-business dependency |
The question legal is actually asking
"Does our data leave our boundary?" is not one question but three, and conflating them produces both false alarms and real breaches. First: does the payload cross into a third party's systems at all? Second: if it does, is it retained, logged, or used to improve their models — and is that contractually excluded? Third: which jurisdiction does it land in, and does that satisfy the residency commitment your own customers were given?
A hosted API answers "yes" to the first and needs written answers to the second and third. That is often perfectly acceptable — a data-processing agreement with retention disabled and a named region is a normal commercial arrangement, and the same argument already applies to the managed database and the email provider you use without a second thought. What is not acceptable is assuming the answer. The most common real-world failure here is not a breach; it is a team discovering after launch that a support-ticket summarizer has been sending regulated content to a region their own contracts prohibit.
Self-hosting genuinely removes this question, which is why it is the honest reason to choose it. But notice what it does not remove: you now own the weights, and model artifacts trained on customer data are themselves classified assets. Moving the boundary is not the same as removing the obligation. See Infrastructure Trust Boundaries and Key Management and Encryption at Rest for what the boundary is made of.
Where the crossover actually is
The economic argument for self-hosting is a crossover: usage-shaped pricing has a slope, dedicated capacity has a floor, and above some volume the floor wins. The mistake teams make is comparing the token price against the instance price and stopping there. The instance is not the cost of self-hosting; the cost of self-hosting is the instance divided by your actual utilization, plus a platform engineer's time, plus the redundancy you need so a device failure is not an outage, plus the capacity you hold for peak.
Do that arithmetic honestly and the crossover moves a long way to the right. A pool at 20% utilization with a warm standby is paying roughly ten times its nominal rate per useful request. Sustained high utilization is what makes self-hosting cheap, and sustained high utilization is exactly what a young product does not have — its traffic is spiky, its volume is uncertain and its model choice is still changing.
Two more factors belong in the decision and rarely make it in. Model *churn*: a hosted API upgrades under you, which is both a benefit (you get better models for free) and a liability (behaviour changes without a deploy, so you need evaluations pinned to a version). And the *exit*: a hosted API behind your own thin interface is genuinely swappable in an afternoon, whereas a self-hosted stack you have tuned for eight months is not. Reversibility is worth real money early, and this is one of the few places where the cheap option is also the reversible one.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Four categories: hosted model API, managed inference, self-hosted inference, dedicated cluster — differing in where the responsibility line falls.
- The hosted API is the correct default, because it costs nothing at idle and is reversible; the others need a named reason.
- The three legitimate reasons to move right: a data-boundary requirement, sustained volume past the crossover, or a model nobody hosts.
- "Does our data leave the boundary?" decomposes into crossing, retention and jurisdiction — answer all three in writing before launch.
- The self-hosting crossover is much further right than the sticker prices suggest, once utilization, redundancy and engineering time are included.
- A hosted API upgrades under you: that is free improvement and uncontrolled behaviour change, so pin evaluations to a version.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Hosted API: the application holds an API credential, calls an external endpoint over TLS, and pays per token; the provider owns model, hardware and scaling.
- • Managed inference: you upload or reference a model artifact, declare an instance type and a scaling policy, and the provider stands up an endpoint inside your account.
- • Self-hosted: your container image bundles an inference server, requests an accelerator, loads weights at startup and serves inside your own network.
- • Dedicated cluster: the same, plus a scheduler, a capacity commitment, multi-tenancy between teams and a queue for training jobs.
- • In every case the application should call through one internal interface, so the choice is a configuration change rather than a rewrite.
- • Whichever you pick, you still own prompt and version pinning, evaluation, rate limiting, retries with backoff, timeouts and a fallback path.
- • You own the data-processing agreement, the retention setting and the region selection — these are configuration, and defaults are frequently not what you want.
- • You own cost attribution per tenant and per feature; no provider does this for you at the granularity finance will ask for.
- • On managed inference and above, you own model artifact versioning, promotion between environments and rollback.
- • On self-hosted and above, add drivers, batching configuration, capacity planning, redundancy and on-call.
- • Provider outage or rate limit on a hosted API: every request fails at once, and without a fallback the feature is simply down.
- • Silent model version change altering behaviour with no deploy, caught by evaluations if you have them and by customers if you do not.
- • Managed endpoint scale-out cold start: a traffic spike queues for minutes while a new instance loads weights.
- • Self-hosted single device failure taking the feature offline, because redundancy was deferred as a cost saving.
- • A migration between options that was never rehearsed, discovered mid-incident to be a two-week project rather than a config flag.
- • Regional misconfiguration sending regulated payloads outside the permitted jurisdiction — a compliance incident, not a technical one.
- • Hosted API scales effortlessly until the account rate limit, which is then a hard wall you negotiate rather than provision around.
- • Managed inference scales on your policy, bounded by cold start; the same weight-loading delay as self-hosting applies.
- • Self-hosted scales as fast as you can obtain accelerator capacity, which in constrained regions is not a given.
- • Across all options, caching and request deduplication buy more headroom than capacity does, because near-duplicate requests are common.
- • Cost scales linearly with usage on the left and steps discontinuously on the right — a useful property when forecasting.
- • The hosted API credential is a spending credential: scope it per environment, rotate it, and alert on spend rather than only on errors.
- • Payloads to a third party need a data-processing agreement, retention disabled where required, and a pinned region — technical controls cannot substitute for the contract.
- • Self-hosting moves the boundary inward but adds the model artifact as a classified asset with its own access control and encryption requirements.
- • A managed or self-hosted inference endpoint must still authenticate its callers; an unauthenticated internal endpoint is a denial-of-wallet and data-access hole.
- • Log what was sent, not necessarily the contents: prompt logs are customer data and inherit its classification.
- • Usage-shaped on the left, fixed-shaped on the right; the crossover depends almost entirely on sustained utilization.
- • Idle is free on a hosted API and fully charged everywhere else — the dominant factor for spiky or immature workloads.
- • Redundancy doubles the self-hosted floor before it serves a single extra request.
- • Engineering time is the line item that never appears on the bill and frequently exceeds it.
- • Caching cuts every option's cost, and it is the cheapest optimization available in all four.
- • Requests, errors and latency split by provider or endpoint, so a degradation is attributable rather than mysterious.
- • Cost per request and per tenant, trended — the number that tells you whether the crossover has arrived.
- • Rate-limit rejections and retry counts, which are the leading indicator of the hosted-API ceiling.
- • Model version in use, recorded per request, so a behaviour change can be correlated with an upgrade.
- • The signal that lies: your own service's health checks and CPU, which stay green through a total provider outage because nothing local is unwell.
- • No model at all. A large fraction of "AI features" are a rule, a lookup table or a regex, and those have no failure modes worth this discussion.
- • A smaller model on ordinary CPU instances, which sidesteps the whole comparison for classification and reranking workloads.
- • A hosted API behind your own thin interface, with self-hosting kept as a documented future option rather than a current project. This is the right first move for almost every team.
- • Caching plus batching against a hosted API often removes the volume argument for self-hosting entirely, at a fraction of the effort.
- • Hosted API buys zero operational burden and reversibility; costs a third-party data boundary, a rate limit you do not control and behaviour that can change under you.
- • Managed inference buys model control without hardware ownership; costs idle billing and provider-specific deployment mechanics.
- • Self-hosting buys the data boundary and unit economics at volume; costs drivers, capacity risk, redundancy and a permanent operational commitment.
- • A dedicated cluster buys the lowest marginal cost at high utilization; costs a platform team and a capacity commitment made before you know the demand.
What people believe, and what is true
Self-hosting is cheaper once you have real volume.
Only at sustained high utilization. Divide device-hours by requests actually served, add redundancy and engineering time, and the crossover is much further out than the sticker comparison suggests.
Using a hosted API means we have no control.
You control version pinning, retries, timeouts, fallbacks, rate limiting, caching, retention settings and region. What you do not control is the hardware — which for most teams is the part they did not want.
Self-hosting solves the compliance question.
It removes the third-party hop. It adds the model artifact, the prompt logs and the inference endpoint as assets you must classify, encrypt and access-control yourself.