The question this answers
Who divides one physical machine among many tenants, and what does sharing that machine cost me?
A workload must run on hardware it does not own, alongside other companies' workloads, with predictable enough performance to hold a latency objective and a boundary strong enough that a bug in a neighbour's software is not an incident in yours.
Scheduled access to real cores, memory and I/O with a per-guest kernel and a per-guest memory map, so that correctness is isolated absolutely while performance is isolated only approximately — and knowing which is which is the point of this lesson.
Two placements, one job
A hypervisor multiplexes four things: CPU time on physical cores, physical memory, block I/O against real devices, and network I/O through a virtual switch. Everything else it does is bookkeeping in service of those four. It is worth holding that list in mind, because every performance surprise on shared infrastructure is contention for one of them, and the four fail in noticeably different ways.
A type 1 hypervisor runs directly on the hardware: it is the lowest software layer, and the guests sit on top of it. This is what every cloud provider runs, because there is no host operating system underneath to add overhead, attack surface, or a second scheduler making decisions about your guest's cores. A type 2 hypervisor runs as an application on an ordinary desktop operating system, using it for drivers and scheduling. That is what a developer laptop runs, and the difference in overhead and isolation is exactly the difference between "the OS underneath is a thin, purpose-built layer" and "the OS underneath is also running a browser".
The distinction matters less as a taxonomy than as a way of reading claims. Benchmarks taken on a type 2 hypervisor on a laptop tell you very little about the same guest on a provider's type 1 host, and isolation arguments made about a laptop VM do not carry either. When someone says "we tested it in a VM", the useful follow-up is which kind, and what else was running underneath.
| Dimension | Type 1 (bare-metal) | Type 2 (hosted) | Container (shared kernel) |
|---|---|---|---|
| Runs on | The hardware itself | A general-purpose host OS | A kernel shared with every other container on the host |
| Kernels running | One per guest | One per guest, plus the host OS | One, total |
| What separates tenants | Hypervisor + CPU virtualization extensions | Hypervisor + host OS, which is also doing other things | Kernel namespaces and cgroups — one kernel bug away |
| Boot time to usable | Tens of seconds | Tens of seconds | Milliseconds to seconds |
| Memory overhead per workload | A whole guest OS | A whole guest OS | Effectively none beyond the process |
| Density on one host | Tens | A handful | Hundreds |
| Suitable for untrusted code | Yes — the ordinary answer | Not for production | Only with extra hardening, and still weaker |
| Where you meet it | Every cloud instance you have ever launched | Your laptop | Your image, and every orchestrator |
Correctness is isolated; performance is not
The hypervisor guarantees that a neighbour cannot read your memory. It does not guarantee that a neighbour cannot make your requests slow. Cores are oversubscribed, memory bandwidth and last-level cache are genuinely shared, and the disk queue on the host is finite. When a co-tenant starts a large build, a backup, or a badly written benchmark, your guest waits — and the waiting is invisible to almost every metric you are looking at.
The tell is steal time: the fraction of the interval during which your vCPU was runnable but not scheduled on a physical core. It is one of the few numbers that lets a guest see the host. Everything else your monitoring reports is measured from *inside* the guest, and from inside the guest, time spent not being scheduled looks like time that simply did not exist. Utilisation stays flat. Load average may rise. Latency doubles.
This is the failure mode worth memorising, because the second-order effect is worse than the first. An autoscaling policy targeting 70% CPU will not fire, because guest CPU has not moved. A health check with a two-second timeout may start failing, so instances are removed from rotation and replaced — with new instances scheduled onto the same contended host family, in the same zone, during the same event. The system responds to a resource it does not have with a mechanism that cannot obtain it.
- Steal time is the guest-visible evidence of host contention. If you are not collecting it, you cannot diagnose this class of incident at all.
- Memory bandwidth and cache contention do not show up as steal time and are effectively invisible from inside the guest — you infer them from latency that has no other explanation.
- Burstable instance families add a second, self-inflicted version of the same shape: credits exhaust and the guest is throttled to a baseline while utilisation looks fine.
- Live migration moves a running guest to another host to let the provider service hardware; it usually shows up as a short latency blip and occasionally as a dropped long-lived connection.
- Dedicated hosts and larger instance sizes reduce this by giving you a larger share — sometimes all — of the physical machine, and they cost accordingly.
$ mpstat -P ALL 5 1 # 09:00, ordinary morning
CPU %usr %sys %iowait %steal %idle
all 31.2 6.4 0.9 0.4 61.1
$ mpstat -P ALL 5 1 # 14:05, during the incident
CPU %usr %sys %iowait %steal %idle
all 28.7 5.9 1.1 22.6 41.7
^^^^
What the dashboards said at 14:05
---------------------------------
guest CPU utilisation ........ 34% (unchanged since 09:00)
memory used .................. 61% (unchanged)
disk queue depth ............. 0.8 (unchanged)
p99 request latency .......... 180 ms -> 1.4 s
autoscaling (target 70% CPU) .. did not fire
health check (2 s timeout) .... 3 instances failed, replaced
replacements ................. same family, same zone, same contention
The only number that moved for a reason: %steal.Why the VM boundary is stronger than the container boundary
A container is a process on the host kernel with its view of the world narrowed by namespaces and its resources capped by cgroups. That is a real boundary and it is enough for the ordinary case: your own services, built by your own pipeline, running next to each other. But the attack surface between two containers is the entire system-call interface of one shared kernel — hundreds of calls, an enormous amount of code, and a steady supply of privilege-escalation findings. If a container escapes, it escapes onto a host that is running everyone else's containers.
Between two guests on a hypervisor, the surface is much smaller: privileged operations trap into the hypervisor, memory translation is nested so a guest cannot even name host-physical memory, and device access goes through a narrow paravirtualized interface. The code that has to be correct is a fraction of the size, and the CPU itself participates in enforcing it. That is why the practical rule is stated the way it is: containers separate your workloads from each other; hypervisors separate you from other companies.
The honest qualifications matter. A hypervisor boundary is not perfect — cross-VM side channels through shared caches and speculative execution are real, which is why providers offer dedicated hardware for workloads that care. And containers running on per-tenant VMs, which is how every managed Kubernetes offering is actually built, inherit the VM boundary for cross-tenant isolation and use the container boundary only within your own trust domain. The two are layered in practice, not opposed. Where the choice is genuinely yours is untrusted code: build steps from customers, notebook execution, plugin runtimes, agent tool sandboxes. There, reach for a VM or a microVM — see Containers vs Virtual Machines and Infrastructure for Model and Agent Workloads.
Key points
- A hypervisor multiplexes four things — CPU time, memory, block I/O and network I/O — and every shared-host performance surprise is contention for one of them.
- Type 1 runs on the hardware and is what providers use; type 2 runs on a host OS and is what your laptop uses. Do not carry conclusions between them.
- Correctness is isolated absolutely; performance is isolated only approximately. That gap is the noisy-neighbour problem.
- Steal time is the guest's only direct view of host contention, and utilisation-based autoscaling is blind to it.
- The VM boundary is stronger because the surface is a narrow, hardware-assisted interface rather than an entire shared syscall API.
- Containers separate your workloads from each other; hypervisors separate you from other companies. In practice they are layered, not opposed.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The hypervisor gives each guest a set of vCPUs, which are scheduled onto physical cores like threads — so a vCPU is an entitlement to time, not a core.
- • Guest memory is mapped through a second translation stage the guest cannot see or address, which is what makes cross-guest memory access impossible rather than merely forbidden.
- • Privileged instructions in the guest trap to the hypervisor, which emulates them; hardware virtualization extensions make the common cases cheap.
- • Storage and network devices are paravirtualized: the guest driver posts descriptors to a shared ring buffer that the host consumes, avoiding full device emulation.
- • When physical cores are oversubscribed, a runnable vCPU that is not currently scheduled accrues steal time, which the guest kernel reports.
- • Live migration copies guest memory to another host while it runs, then pauses briefly to transfer the remainder and resume — visible as a short stall.
- • Collect steal time, or its provider equivalent, on every instance — it is not in most default dashboards and it is the only signal for this failure class.
- • Know which of your instance families are burstable and what happens when credits run out, because that failure looks identical to neighbour contention.
- • Choose instance sizes with contention in mind: larger sizes generally mean a larger, more predictable share of a host.
- • Handle live-migration and scheduled-maintenance events: drain the instance, or at minimum make the workload tolerant of a short stall and a dropped connection.
- • For latency-sensitive or licence-bound workloads, evaluate dedicated hosts explicitly rather than discovering the need during an incident.
- • Tail latency doubles while every guest-side utilisation metric stays flat, because the vCPUs are runnable and not running.
- • Autoscaling does not fire during the exact event that needed it, because the signal it watches — guest CPU — is unaffected by contention.
- • Health checks with tight timeouts fail during contention, so healthy instances are replaced with new ones landing on the same contended hardware.
- • A burstable instance exhausts its credits mid-incident and is throttled to a fraction of its nominal capacity, and the graph looks like a plateau nobody can explain.
- • A live migration stalls a guest long enough to break a long-lived connection or trip a leader-election timeout in a clustered service.
- • A workload running untrusted customer code on shared-kernel containers is compromised through a kernel vulnerability that a VM boundary would have contained.
- • Adding more small instances on the same contended family and zone does not solve contention; it distributes it and pays more for the privilege.
- • Larger instance sizes are the more reliable answer to neighbour contention, up to a dedicated host — which converts a variable performance problem into a fixed cost.
- • Density is the provider's lever, not yours: oversubscription ratios are not published and change, so treat consistent performance as something to verify rather than assume.
- • For latency objectives measured at the tail, contention is frequently the binding constraint long before the application is.
- • Cross-guest memory access is prevented structurally by nested translation, not by a permission check that could be misconfigured.
- • The residual risk is side channels through shared caches and speculative execution, which is what dedicated hardware and per-tenant hosts exist to address.
- • A shared-kernel container boundary is weaker in kind, not merely in degree: the exposed surface is the whole syscall interface of a kernel that other tenants also depend on.
- • Managed orchestrators run your containers on nodes that are your VMs, so cross-tenant isolation is still the hypervisor's job — see Infrastructure Trust Boundaries.
- • Untrusted or customer-supplied code is the clearest case for a VM or microVM per unit of work, and it is how function platforms are built internally.
- • Shared tenancy is the default and the cheapest; you are paying partly in performance variance.
- • Dedicated hosts and metal instances remove neighbour contention and cost substantially more per unit of compute — a real trade, not a premium for nothing.
- • Burstable families are cheap because they sell you a baseline with a credit allowance, which is excellent for genuinely bursty low-average work and dangerous for steady load.
- • Contention has an indirect cost: instances replaced by failing health checks, and capacity added by autoscaling that was never the constraint.
- • Steal time per instance, alerted on, and correlated with latency rather than looked at in isolation.
- • Burst credit balance on burstable families, which is the self-inflicted twin of the same problem.
- • Provider maintenance and migration events on the instance, so a mysterious stall has a documented cause.
- • Tail latency alongside utilisation. The pairing is what makes contention visible: latency up, utilisation flat, steal up.
- • The signal that lies: average guest CPU utilisation. During contention it is stable and reassuring and completely irrelevant, and it is the number most autoscaling policies are built on.
- • Do nothing and accept variance. For batch, asynchronous or internal workloads with no tail-latency objective, neighbour contention is a curiosity rather than a problem, and paying to remove it is waste.
- • A larger instance size before a dedicated host: it is a smaller step, usually enough, and does not commit you to a fixed capacity purchase.
- • A managed platform, which makes contention the provider's problem to hide — you lose the diagnostic view, which is a genuine loss when latency does move.
- • Containers on shared hosts, when everything on the host is yours. The stronger boundary is only worth its overhead when the code on the other side is not under your control.
- • On-premises hardware, when licensing is billed per physical core or the workload needs direct device access. See On-Premises vs Cloud for the honest version of that comparison.
- • The hypervisor boundary buys cross-tenant safety and charges a whole guest OS per workload, plus tens of seconds of start time.
- • Shared tenancy buys a much lower price and charges performance variance you cannot control and can barely see.
- • Dedicated capacity buys predictability and charges a fixed bill whether the workload uses it or not.
- • Containers buy density and speed and charge you a boundary that is only appropriate for code you trust.
What people believe, and what is true
A vCPU is a CPU core.
A vCPU is an entitlement to scheduled time on a core, usually a hardware thread, on a host that is oversubscribed by a ratio the provider does not publish.
If my CPU utilisation is normal, the instance is fine.
Utilisation is measured inside the guest and excludes time the guest was not scheduled. Contention presents as flat utilisation and doubled latency — check steal time.
Containers and VMs are just two ways of doing the same thing.
They isolate different things at different strengths. Containers narrow a process's view of one shared kernel; hypervisors give each guest its own kernel behind a hardware-assisted boundary.
Noisy neighbours are a myth from the early cloud.
Providers have got much better at hiding it, and it still happens. The reason it is rarely diagnosed is that the metric that shows it is not on the default dashboard.