The question this answers
What am I actually buying when I choose an instance size, and which of its dimensions will constrain my workload first?
An API process must have enough CPU to meet a latency target, enough memory to hold its working set without being killed, enough local disk for its temporary files and logs, and enough network throughput to serve responses and reach its database.
A defined resource envelope: a stated number of virtual CPUs, a fixed amount of memory, a local disk with a stated size and performance, and a network allocation — all enforced, all metered, and all finite.
Four dimensions, and the one you look at is rarely the one that runs out
Every compute offering, on every provider and on-premises, resolves to the same four dimensions plus two qualifiers. The four: CPU, memory, local storage, network. The two qualifiers: processor architecture, and the lifecycle guarantee — whether the instance is yours until you release it, or whether it can be reclaimed with two minutes' notice.
Teams size by CPU because CPU is the number in the instance name. In practice the constraint is usually somewhere else. A JSON-heavy API is typically memory-bound before it is CPU-bound, because every in-flight request holds a parsed object graph. A log-shipping agent is network-bound. A CI runner is disk-bound. And a workload with a large per-process heap running many processes hits the memory ceiling at a CPU utilization that looks embarrassingly low — at which point adding CPU is the intuitive move and does nothing at all.
The other trap is that these dimensions are coupled in the catalogue. You do not buy memory; you buy an instance type with a fixed CPU-to-memory ratio. Needing 2 vCPU and 32 GB means buying an 8 vCPU instance in a memory-optimized family and leaving six cores idle, or accepting OOM kills. That coupling is the single biggest driver of the gap between what a workload needs and what it is billed for — the subject of Right-Sizing Without Causing an Outage.
| Dimension | What consumes it | What it looks like when it runs out | What you actually do |
|---|---|---|---|
| CPU | Request handling, serialization, TLS, compression, garbage collection | Latency rises smoothly with load; run-queue length grows before utilization pins at 100% | Add instances, or profile — a single hot function is common and cheaper to fix than to out-scale |
| Memory | Working set, per-request object graphs, caches, connection buffers, runtime heap overhead | The process disappears. No stack trace, no application log — an OOM kill is a signal, not an exception | Raise the limit or cut the working set. Adding instances does not help a per-process ceiling |
| Local storage | Temporary files, logs, build artifacts, container image layers | Writes fail. The application returns errors while CPU and memory look perfectly healthy | Rotate logs, move artifacts off-instance, mount network storage for anything that must persist |
| Network | Response bodies, database traffic, object-storage transfers, replication | Throughput plateaus at the instance ceiling; latency rises with no CPU correlation | Larger instance class (bandwidth usually scales with size), or move the bytes closer |
| Architecture (x86 / ARM) | Chosen once; determines which images and native dependencies run at all | The container will not start, or a native library is missing — a build problem, not a runtime one | Build multi-architecture images. ARM is frequently cheaper per unit of work if your stack supports it |
| Lifecycle guarantee | On-demand, reserved, or interruptible/spot capacity | Interruptible instances vanish with a short warning; the workload must be able to lose one mid-task | Match the guarantee to the workload — see The Instance Lifecycle |
A vCPU is a share, not a core
A virtual CPU is a scheduling entity presented by a hypervisor, usually mapped to one hardware thread rather than one physical core. On a machine with simultaneous multithreading, two vCPUs frequently share the execution resources of one physical core, so 2 vCPU is not twice the throughput of 1 vCPU for a compute-bound workload. This is not deception; it is what the unit means, and it is why the honest way to size is to measure your workload rather than to reason from the number.
The more consequential detail is steal time: the fraction of time a virtual CPU was ready to run and the hypervisor gave the physical resource to someone else. On dedicated capacity it sits at zero. On shared or burstable capacity it can rise sharply, and when it does, the application is slow while every in-instance metric looks unremarkable. Steal time is the only signal available from inside the guest that says "the problem is a layer you do not own" — see Hypervisors and Shared Hosts and The Infrastructure Layer Stack.
Burstable instance families add a third mechanism: a credit balance that accrues while you are below a baseline and is spent while you are above it. They are excellent for genuinely spiky low-average workloads and disastrous for a steady one, because when credits are exhausted the instance is throttled to its baseline — often a small fraction of the vCPU count in its name — and the resulting latency cliff looks exactly like an application regression.
- A vCPU is usually a hardware thread. Two vCPUs often share one physical core, so throughput does not double.
- Steal time is the only in-guest evidence that the hypervisor layer is the problem, and it is almost never on a default dashboard.
- Burstable families trade a credit balance for a low price; exhausting credits produces a latency cliff that looks like a code regression.
- Run-queue length is a better saturation signal than CPU utilization, because utilization saturates at 100% and stops telling you how bad it is.
$ lscpu | head -8 Architecture: x86_64 CPU(s): 4 <- 4 vCPUs Thread(s) per core: 2 <- so ~2 physical cores of execution resource Core(s) per socket: 2 Hypervisor vendor: KVM <- you are a guest; something schedules you $ vmstat 1 3 procs -----------cpu------------ r b us sy id wa st 6 0 41 6 9 0 44 <- st = 44% STEAL 7 0 39 7 11 0 43 you are ready to run and not running 6 0 43 5 9 0 43 in-guest "utilization" looks ~50% WHAT THIS MEANS run-queue (r) = 6 on 4 vCPUs -> already queueing steal 43% -> nearly half your scheduled time went elsewhere a dashboard showing "50% CPU" here is telling the truth and the truth is useless: the instance is starved, not idle. BURSTABLE FAMILIES: when the credit balance hits zero the same instance is capped at its baseline (often 10-40% of one vCPU per vCPU in the name). Latency cliffs, no code change, no deploy. ILLUSTRATIVE: invented output to show which counters matter.
The compute bill has a shape, and it is mostly not usage
Compute is billed per instance-second or instance-hour for as long as the instance exists — not for as long as it is doing work. An instance at 4% utilization costs exactly what an instance at 94% costs. That single fact is why Idle Capacity: Headroom or Waste? is a first-class topic and why utilization is a cost metric before it is a performance metric.
Three modifiers change the shape substantially. Commitment (reserved capacity or a spend commitment) trades flexibility for a materially lower rate on steady workloads. Interruptible capacity trades a survivability guarantee for a much lower rate, which is close to free money for batch and CI and unusable for a stateful primary. Architecture choice can shift price-per-unit-of-work meaningfully where your stack supports ARM builds.
The line teams do not predict is attached storage and data transfer. The instance rate covers the instance; the network-attached root volume, its provisioned IOPS, its snapshots, and every gigabyte that crosses a zone or leaves for the internet are separate meters. A fleet that looks cheap by instance-hour can be dominated by the volumes and the traffic attached to it.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Compute is four dimensions — CPU, memory, local storage, network — plus architecture and a lifecycle guarantee.
- The dimension that runs out first is usually not CPU, and the failure looks different for each: latency for CPU, a vanished process for memory, write errors for disk, a throughput plateau for network.
- Instance families couple the dimensions in fixed ratios, which is the main reason billed capacity exceeds needed capacity.
- A vCPU is a scheduled share of a hardware thread; steal time is the only in-guest signal that the layer below you is the problem.
- You pay for the instance existing, not for it working, so utilization is a cost metric before it is a performance one.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The provider partitions a physical host with a hypervisor and presents each guest a fixed number of virtual CPUs and a fixed memory allocation.
- • Virtual CPUs are scheduled onto physical threads; contention shows in the guest as steal time and in latency, not in the guest's own utilization figure.
- • Memory is generally a hard allocation — exceeding it results in a kill rather than in degradation, which is why memory failures are abrupt and CPU failures are gradual.
- • Network bandwidth is capped per instance and usually scales with instance size, which is why a network-bound workload is sometimes fixed by a larger instance rather than by more of them.
- • The root volume is typically network-attached storage, so "local disk" latency is a network characteristic — see Block Storage.
- • Choosing the instance family and size, and revisiting it — nothing right-sizes for you and the first guess is nearly always wrong in at least one dimension.
- • Watching for the coupled-ratio trap: paying for six idle cores to get the memory you needed is a design smell with a monthly cost.
- • Keeping the guest OS and its packages patched on anything you own down to that layer — see The VM Lifecycle.
- • Deciding the lifecycle guarantee per workload, and making interruptible workloads genuinely interruptible rather than hoping.
- • Managing local disk: logs and temporary files fill it, and a full disk is an outage with healthy-looking CPU and memory.
- • OOM kill: the process is terminated by the kernel with no application-level error, and the only evidence is a system event nobody was watching.
- • CPU steal from a noisy neighbour: latency doubles while in-guest utilization looks moderate and nothing in the application changed.
- • Burstable credits exhausted: a hard latency cliff on a steady workload, indistinguishable from a code regression without the credit metric.
- • Local disk full from unrotated logs: writes fail, the application errors, and every dashboard except disk usage looks fine.
- • Network ceiling reached: throughput plateaus and latency rises with no CPU correlation, commonly on a small instance doing large object transfers.
- • Vertical scaling is immediate and bounded by the largest instance available; horizontal scaling is unbounded and requires the workload to be stateless — see One Big VM or Several Small Ones.
- • Memory is the dimension that most often forces vertical scaling, because a per-process working set does not shrink by adding instances.
- • Network bandwidth generally scales with instance size, which makes it one of the few genuine reasons to size up rather than out.
- • The real ceiling at scale is frequently a provider quota on instances of a given family in a given zone, met at the worst possible moment.
- • Multi-tenancy is real: your guest shares physical hardware with other tenants, mediated by the hypervisor — a strong boundary, and not an infinite one.
- • Dedicated hosts or bare metal remove the co-tenancy question at a substantial price premium, which regulated workloads sometimes pay.
- • Local disk on an instance persists until the instance is destroyed; anything sensitive written there needs encryption and a deliberate destruction path.
- • An instance's attached identity is reachable from any process on it, so a compromised process inherits the instance's permissions in full — see Roles vs Static Keys.
- • Instance-hours dominate and are independent of utilization, which makes idle capacity the largest recoverable waste in most accounts.
- • Attached storage bills on provisioned size, not on bytes used, so over-provisioned volumes cost exactly as much as full ones.
- • Commitment discounts are the largest available lever on a steady baseline, at the cost of flexibility for one to three years.
- • Data transfer out is the meter that grows with users rather than with instances, and it is the one that outgrows compute first.
- • Per-dimension saturation, not one CPU number: run-queue length, memory headroom, disk usage and free space, and network throughput against the instance ceiling.
- • Steal time and, on burstable families, the credit balance — both are absent from default dashboards and both explain otherwise inexplicable latency.
- • OOM kill events from the platform or kernel, since the application cannot report its own termination.
- • The signal that lies: average CPU utilization. It hides steal, hides throttling, hides a single saturated core in a multi-core instance, and is the number people size on.
- • A managed container service or a function platform, where you declare CPU and memory per task and never choose an instance family at all. For most application workloads this removes the coupled-ratio problem entirely.
- • A single larger instance instead of a fleet, for a workload that is genuinely stateful or memory-bound — one machine has no distribution problem.
- • Interruptible capacity for anything batch-shaped: CI, data processing, rendering, model training. It is the largest single cost lever available and costs nothing but a retry mechanism you should have anyway.
- • Not sizing at all yet: run the workload on something plausible for a week and size from measurements, rather than reasoning from the instance catalogue.
- • Larger instances buy headroom in every dimension at once and charge for the dimensions you did not need — the coupled-ratio tax.
- • Burstable and interruptible families buy a much lower rate and charge a performance cliff or an eviction, both of which must be designed for.
- • Commitment buys a lower rate and charges flexibility, which is a bad trade for a workload whose shape you do not yet know.
What people believe, and what is true
More vCPUs means proportionally more throughput.
vCPUs are usually hardware threads sharing physical cores, and many workloads are bound by memory, network or a single hot code path. Measure before you scale a dimension.
The instance is idle, so it is cheap.
Billing is for existence, not for work. Idle instances are the most common and most recoverable waste in a cloud account — see Idle Capacity: Headroom or Waste?.
High CPU means we need a bigger instance.
It might mean one function is doing something quadratic. Profiling costs an afternoon; a permanently larger fleet costs every month forever.