Cloud ML Services
Every managed ML service is an implementation of a primitive you should already be able to name — a training job, a GPU, a registry, a hosted endpoint. Learn the primitive, then map the vendor.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A cloud provider offers a managed training service, a model hosting service and a feature store. What is each one actually doing underneath, what does it hide, and what do you need to check before trusting it?
We are moving our models to a cloud provider. The console shows a dozen ML products and the account team recommends all of them. Our engineers know how to train a model on a VM and serve it from a container. I want to know which of these products are things we already understand with a different name, which ones hide something we need to see, and how to avoid designing our system around a product that gets renamed next year.
Use the managed service for everything. The provider has solved training, serving and monitoring; wiring them together is faster than building, and the console makes it obvious.
The managed training job is a container on a VM with logging and a lifecycle. It hides the VM, which is fine, and it hides the machine type's memory, which is not: the run fails on a larger dataset with an out-of-memory error surfaced as a generic job failure (Memory Bandwidth & VRAM).
- The managed training job is a container on a VM with logging and a lifecycle. It hides the VM, which is fine, and it hides the machine type's memory, which is not: the run fails on a larger dataset with an out-of-memory error surfaced as a generic job failure (Memory Bandwidth & VRAM).
- The hosted endpoint autoscaled on CPU. The model was GPU-bound and latency-sensitive; CPU never moved, the endpoint never scaled, and the tail latency under load was the first anyone knew (Latency Breakdown).
- The feature store was adopted because it was on the recommended list. It required an online serving layer the team did not need — every model was batch — and its cost was a line item nobody could justify, which is the "feature stores are mandatory" mistake bought rather than built.
- Two years in, the provider renamed and re-tiered the hosting product. The architecture diagram named the product, not the primitive, and the migration plan had to be rediscovered from the console.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The models are unchanged. The target of this decision is a system whose architecture is expressed in primitives — a training job, a batch prediction run, an artifact in a registry — so that a provider's product is a swappable implementation rather than a load-bearing dependency.
- Training runs on a VM, reads from object storage, writes an artifact back. Serving is a container behind a load balancer. Features are computed by a warehouse job. GPU use is one machine, requested by hand.
- The provider's offering, in the categories every provider has: managed training jobs, GPU compute, a model registry, model hosting, batch prediction, a feature store, workflow orchestration and monitoring. Product names differ per provider and change; the categories do not.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Every managed ML service is a primitive plus automation plus a hidden set of defaults. A training job service is: pick a machine, pull a container, mount storage, run, capture logs and exit code, tear down. Model hosting is: a container with a load balancer, a health check and an autoscaler. Batch prediction is: a job that reads a table, runs the artifact, writes a table. A registry is a bucket with metadata and stages. A feature store is a key-value store plus a batch job that fills it (Feature Stores).
- What the service hides is exactly what you need to check: the machine type and its memory, the autoscaling signal, the artifact format the endpoint loads, the timeout, the retry, the IAM identity the job runs as, whether the registry's stages are enforced or advisory (The Model Registry).
- Because the primitive is the same across providers, an architecture stated in primitives survives renames and migrations. An architecture stated in product names is a dependency on a marketing department.
Primitive, what it hides, what to check
The console presents products. The architecture needs primitives. This matrix is the translation: for each category every provider offers, the primitive underneath, the things the managed layer hides that you will need at some point, and what to verify before trusting it in production.
The product names are deliberately absent. They differ by provider, they change, and a lesson that named them would be wrong within the year. Take the current names, quotas and prices from the provider's documentation on the day you need them.
| Category | The primitive | What the managed layer hides | What to check |
|---|---|---|---|
| Training jobs | Run a container on a machine, mount storage, capture logs and exit code, tear down | Machine type and memory; the identity it runs as; retry behaviour; where checkpoints go | OOM behaviour on a large run; least-privilege identity; checkpoint path survives the job |
| GPU compute | A machine with an accelerator, a driver and a quota | Availability per region; preemption; driver and runtime versions; the actual utilisation | Quota before the deadline; utilisation during runs; spot interruption handling (Checkpointing) |
| Model registry | A bucket with versioned artifacts, metadata and stage labels | Whether stages are enforced or advisory; what lineage is recorded; who can promote | A promotion without an evaluation record is rejected, not just discouraged |
| Model hosting | A container behind a load balancer with a health check and an autoscaler | The scaling signal; the request timeout; the artifact format it loads; cold-start time | Scaling signal matches the bottleneck; timeout exceeds tail latency; a fallback exists |
| Batch prediction | A job that reads a table, runs the artifact, writes a table | Output schema; partial-failure semantics; whether a rerun overwrites or appends | Idempotent reruns; schema pinned; downstream consumers named |
| Feature store | A key-value store filled by a batch job, plus an offline table for training | Freshness of the online values; point-in-time semantics of the offline table; cost of the online tier | Whether you need online serving at all; point-in-time correctness on a known example |
| Workflow orchestration | A DAG runner with scheduling, retries and state | Retry semantics per step; what "success" means for a step; backfill behaviour | A non-idempotent step under retry; a backfill over a training step (ML Orchestration) |
| Monitoring | Prediction logs plus distribution computations plus alerts | What is logged by default; whether outcomes can be joined; retention | The join from prediction to outcome to model version actually works on one request |
The endpoint that did not scale
The hosting failure is worth walking through because it is the most common. A hosted endpoint is a container with an autoscaler, and an autoscaler needs a signal. The default is CPU. A model that spends its time on a GPU, or waiting on a feature lookup, does not move CPU, so the autoscaler sees a calm system while the request queue grows.
Nothing in the offline evaluation could have shown this — it is a serving property, not a model property — and nothing in the console showed it until load arrived. The fix is a check on the hidden default: what is the scaling signal, and does it correlate with the bottleneck.
Load test in staging at steady moderate traffic: latency within budget, autoscaler never needed.
At the first peak, tail latency several times the budget; the autoscaler at minimum replicas; the client timing out and showing users a default.
- 1The autoscaling signal was CPU utilisation; the model was GPU-bound, so CPU stayed low while the GPU queue lengthened.
- 2The staging load test never exceeded one replica's capacity, so the scaling path was never exercised.
- 3The request timeout on the endpoint was shorter than the loaded tail latency, converting slow responses into errors.
The identity the job runs as
The least visible hidden default is the identity. A managed training job runs as some principal, and the convenient default is one that can read every bucket in the project, so the demo works. A training script that reads the wrong bucket, or a dependency that has been tampered with, then has access it should never have had.
The primitive here is not an ML primitive at all: it is a workload identity with a policy, the same as any job on the provider. The ML-specific part is that training jobs pull many dependencies and read large datasets, which makes both the exposure and the blast radius larger than a typical service.
The machine memory, scaling signal, timeout, identity and loading format of each managed component are set for this workload, not left at the provider's defaults.
holds when Each component's hidden defaults are listed in the architecture document, checked at deployment, and re-checked when the provider changes product versions.
breaks when A product version changes a default; a new model with a different bottleneck is deployed on the old endpoint configuration; the dataset grows past the job's memory.
respond Fix the default, then add its check to the deployment pipeline so the next product version cannot silently reset it.
1# what the job IS, independent of the provider's product name2training_job:3 container: registry/models/churn-train:3f9a1c24 machine:5 accelerator: gpu # class, count and memory from current provider docs6 memory_gb: 64 # checked against the dataset size, not defaulted7 identity: churn-training-sa # least privilege: read one snapshot, write one run prefix8 inputs:9 dataset: object-store://datasets/churn_training_v14/ # immutable snapshot10 outputs:11 run_prefix: object-store://models/churn/runs/{run_id}/12 checkpoints: object-store://models/churn/runs/{run_id}/ckpt/13 interruption: resume_from_checkpoint # required if the machine is preemptible14 timeout_minutes: 24015 retries: 0 # a training job is not idempotent by defaultEvery line names something the managed layer would otherwise default. The identity and the memory are the two that fail silently: one as an exfiltration path, the other as a generic job failure on the first large dataset.
How to build it
Most important first.
- Name the primitive first for every component: "a training job that runs container X on a GPU machine with Y memory, reading snapshot Z from object storage". Then map to the provider's current product. Keep the primitive in the architecture document and the product in the deployment config.
- For each managed service, write down what it hides and check it: the machine type, the scaling signal, the loading format, the timeouts, the identity. The matrix below is the checklist.
- Adopt a managed capability when you have the need it serves, not because it is on the recommended list. Batch models do not need an online feature store; a single model does not need workflow orchestration.
- Use current provider documentation for vendor-specific details — quotas, pricing, product names, GPU availability — rather than any static source, including this lesson. What is stable is the primitive and the checklist.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For training: cost per run and GPU utilisation during the run. A managed job on an idle GPU is the same waste as a VM on an idle GPU with a nicer console (Training Cost).
- For hosting: tail latency at load and the autoscaling signal's correlation with it. If the signal is CPU and the model is GPU-bound, the correlation is zero.
- Number of products used is not a measure of anything. Number of primitives the team can name for their own system is.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every managed component is documented in the architecture as a primitive with its hidden defaults listed and checked — machine memory, scaling signal, identity, timeout, loading format.
- The provider's product can be replaced with another implementation of the same primitive without changing the model, the artifact or the feature definitions.
- Vendor-specific details in the team's runbooks are dated and re-verified against current provider documentation at each migration or renewal.
- Offline: for each managed service, run the failure it hides on purpose in a staging account — a dataset larger than the job's memory, a load test past the endpoint's scaling signal — and confirm the failure is visible and attributable.
- Online: audit the identity each job and endpoint runs as, against the minimum it needs. Broad defaults are the norm and the finding is usually immediate.
- Over time: once a year, re-map every primitive to the provider's current product and pricing. A component whose product no longer exists is a migration that has not been scheduled.
What can go wrong
- The managed job's default identity has broad storage permissions, so a training script can read any bucket, and a bug or a poisoned dependency exfiltrates data (The Model Supply Chain).
- The hosted endpoint has a request timeout shorter than the model's tail latency and returns errors that the client interprets as "no prediction" without a fallback (Serving Fallbacks).
- Batch prediction writes to a table with the provider's default schema, and a downstream consumer depends on a column name that changes with a product version.
- Naming the primitive and checking the hidden defaults is slower than clicking through the console, and the account team will not do it for you.
- Keeping the architecture provider-neutral costs some of the provider's integration conveniences — the one-click wiring between their registry and their endpoint.
- Checking every hidden default finds problems that then need fixing, which delays the migration the team was measured on.
- "The managed service handles scaling." It scales on the signal it was given. A GPU-bound model behind a CPU-scaled endpoint does not scale; check the signal.
- "Feature stores are mandatory — the provider has one." The provider has one because some customers need online feature serving. A batch-only system does not; a bought feature store is still infrastructure with a bill.
- "We are locked in, so provider neutrality is pointless." Naming the primitive is not about switching providers next year. It is about understanding what you are running, which is what lets you debug it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat every managed ML product is a primitive with automation and hidden defaults holds across providers and years, which is why the primitive is the thing to learn; only the product names and the specific defaults vary.
- FRAMEWORK-SPECIFICWhich defaults are hidden, what the scaling signals are, what artifact formats an endpoint loads and how registry stages are enforced differ by provider and change between product versions — these are exactly the details to take from current provider documentation rather than from this lesson.
- SCALE-SPECIFICA team with one model and moderate traffic gains little from managed training and hosting beyond convenience; a team running many GPU jobs or serving high-variance traffic gains scheduling and autoscaling it would otherwise have to build, and the hidden defaults matter more because the load is higher.
Where the depth lives
This domain teaches the model and hands the rest off by name.