Infrastructure Fundamentals

The Infrastructure Layer Stack

Application → runtime → container or VM → host OS → physical compute → network and storage → data centre. Every incident lives at exactly one layer, and most debugging time is spent looking at the wrong one.

▶ Run the lab

The question this answers

Infrastructure question

What sits between my application code and the electricity, and which layer is a given failure actually coming from?

Application requirement

An engineer is paged for "the API is slow". The application code has not changed. Something below it has, and there are seven candidate layers, each with a different owner and a different diagnostic.

What it provides

A fixed ordering to reason with. Each layer provides a specific guarantee to the one above it and fails in a characteristic way, so a symptom can be attributed to a layer instead of investigated everywhere at once.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

Seven layers, and what each one promises the layer above

The stack below is the domain's spine. It is not an OSI-style formalism; it is a debugging tool and a responsibility map. Read it downward to understand what you are standing on, and upward to understand how a low-level failure will present itself — because it never presents itself as what it is. A saturated disk on the host does not say "disk saturated"; it says "our p99 doubled and the database looks fine".

The critical property is that each layer hides the one below it, and hides it imperfectly. A container promises you an isolated filesystem and process tree, and does not promise you an isolated page cache or an uncontended CPU. A virtual machine promises you a machine, and does not promise you exclusive access to the physical cores — which is what steal time is, and why an instance can be slow while every metric inside it looks idle. Every abstraction leaks, and the leaks are where the interesting incidents come from.

The second property is that the owner changes partway down. Somewhere in this stack is a line where your responsibility ends and the provider's begins, and where that line sits depends on the compute model you chose. That is the subject of Shared Responsibility; the layer stack is what makes the line drawable.

Read down to see what you stand on; read up to see how a failure will present.
Applicationdepth: Software Architecture / API Design
provides Business behaviour: the handler that answers a request, the job that processes a queue message.
fails as An exception, a wrong answer, a slow query, a leaked connection. Visible directly to users and to your own error tracking.
Language runtimedepth: Operating Systems → language runtimes
provides Memory management, threads or an event loop, and the standard library the application assumes.
fails as Garbage-collection pauses that look like network latency, a blocked event loop that looks like a dead process, a heap limit that kills the process with no application error.
Container or VM imagedepth: Operating Systems → namespaces, cgroups
provides The packaged filesystem and dependencies — the promise that what you tested is what runs.
fails as A missing library that only appears in production, a base image with a vulnerable package, an image so large that scale-out is minutes late — see Why Image Size Is an Infrastructure Problem.
Isolation boundary (container runtime / hypervisor)depth: Operating Systems → virtualization, cgroups
provides A private view of process table, filesystem and network, and enforced CPU and memory limits.
fails as OOM-kill with no application stack trace, CPU throttling that shows as latency with low reported utilization, steal time from a noisy neighbour on the same physical host.
Host operating systemdepth: Operating Systems
provides Scheduling, virtual memory, file descriptors, the network stack, the actual system calls.
fails as File-descriptor exhaustion presenting as connection refused, page cache pressure, a full disk that makes writes fail while CPU and memory look healthy.
Physical computedepth: Cloud → [[compute-fundamentals]]
provides Cores, memory, NICs and local devices — the finite thing everything above is dividing up.
fails as Host failure that terminates every instance on it at once, thermal throttling, a hardware fault that presents as random instance termination.
Network and storage fabricdepth: Computer Networking; Cloud → [[storage-fundamentals]]
provides Reachability between hosts, and durable block and object storage that survives the host.
fails as Network-attached disk latency spikes that look exactly like a slow database; a partition inside one zone; a routing or firewall change that makes a healthy service unreachable.
Data centredepth: Cloud → [[failure-domains]]
provides Power, cooling, physical security and the building itself — the reason Regions and Availability Zones exists.
fails as A zone-level event. Everything above it fails simultaneously, which is the one failure mode that is unambiguous.

A symptom, read down the stack

Take the page: "checkout p99 went from 180 ms to 2.4 s, no deploy". The layered stack turns an unbounded search into an ordered one, and the order matters because the cheap checks are at the top and the expensive ones are at the bottom.

Application: did anything change — a feature flag, a config value, a data volume crossing a threshold that flipped a query plan? Runtime: are GC pauses or event-loop lag correlated with the latency? Container: is the process being CPU-throttled against its limit, which shows as latency while utilization looks moderate — see OOM Kills and CPU Throttling. Host: are file descriptors, connections or disk space near a limit? Compute: is steal time non-zero, meaning another tenant is taking cycles? Fabric: has network-attached disk latency risen, or is one zone's path degraded?

The reason to walk this deliberately is that the top and bottom layers produce *identical* symptoms. Elevated latency with normal CPU is the signature of a slow query, a GC pause, CPU throttling, disk latency and a network partition. Five layers, one symptom. Without a layer model you check whichever one you know best, which is why teams with strong application skills spend hours in the profiler while the answer is a saturated network disk.

  • Elevated latency with normal CPU is produced by at least five different layers. The metric does not identify the layer; the ordered walk does.
  • A container reporting 60% CPU while being throttled is not lying — it is reporting its own consumption, not its quota denial. Different counter, different layer.
  • Steal time is the only in-instance signal of a layer you do not own, which makes it disproportionately valuable and almost never on a dashboard.
  • Network-attached storage latency is the most commonly missed cause of "the database got slow", because the database is innocent and its own metrics say so.
One symptom, six candidate layers — check in order, top is cheapest
123456p99 2.4 s no deployApp: query plan? flag? data volume?Runtime: GC pause? event loop blocked?Container: CPU throttled against its limit?Host OS: fd limit? disk full? conntrack?Compute: steal time? noisy neighbour?Fabric: disk latency? zone path degraded?
ClientGateway / LBServiceWorkerDatabaseCacheQueue / LogObject storageCDNExternal system

Where the abstraction leaks, and who owns the leak

Every layer boundary is a contract, and each one has a well-known clause that is not honoured the way people assume. Knowing these four or five leaks is most of what separates an engineer who can debug production from one who can only debug an application.

The other axis is ownership, and it moves with the compute model. On a virtual machine you own everything from the guest OS upward; on a managed container platform you own from the image upward; on a function platform you own from the code upward. Choosing a compute model is therefore choosing where in this stack your pager stops — which is the real content of Choosing a Compute Model, and why "serverless has no servers" is a statement about ownership rather than about physics.

The corollary is uncomfortable and worth stating: the layers you no longer own can still take you down, and you will have less visibility into them precisely because you handed them over. A managed platform's scheduler making a bad placement decision is invisible to you in a way that your own scheduler making the same decision would not be.

BoundaryWhat it promisesThe leakWho owns it on VM / container platform / functions
App ↔ runtimeMemory is managed for youGC pauses are latency your code did not cause and cannot see in a profiler sampleyou / you / you
Runtime ↔ imageWhat you tested is what runsBase image drift and unpinned dependencies — the image is only reproducible if you made it soyou / you / you
Image ↔ isolationYou get CPU and memory limitsExceeding memory is an OOM kill with no application error; exceeding CPU is throttling, not an error at allyou / shared / provider
Isolation ↔ hostA private view of the machinePage cache, kernel version, conntrack tables and file descriptors are shared or host-wideyou / provider / provider
Host ↔ physicalA machine of a given sizevCPUs are shares of physical cores; steal time is the leak — see Hypervisors and Shared Hostsprovider / provider / provider
Physical ↔ fabricA disk and a network interfaceBoth are network services with their own queues, and a "disk" can have a bad minute for reasons inside the storage fleetprovider / provider / provider
The leak at each boundary, and where ownership sits per compute model.

Key points

  • Seven layers: application, runtime, image, isolation boundary, host OS, physical compute, and network/storage fabric on top of a data centre.
  • Each layer hides the one below it imperfectly; the imperfections — GC pauses, throttling, steal time, network-disk latency — are where production incidents come from.
  • One symptom maps to several layers. An ordered walk from cheapest to most expensive check is what turns an unbounded search into a bounded one.
  • The compute model decides where in this stack ownership transfers to the provider, which is the same thing as deciding where your pager stops.
  • Layers you no longer own can still take you down, and you have less visibility into them because you handed them over.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • Physical hardware in a facility is divided by a hypervisor into virtual machines, each with a guest kernel that believes it owns a machine.
  • Inside a guest, a container runtime uses kernel namespaces and cgroups to give a process group a private filesystem and process view plus enforced resource limits.
  • Inside a container, a language runtime maps application objects onto that limited memory and those scheduled cores.
  • Storage and network are not local devices but fabric services reached over the network, which is why their latency behaves like a network dependency rather than like a bus.
What you still own
  • Metrics from at least three layers — application, container/instance, and host or fabric — or a symptom at one layer cannot be excluded from the others.
  • Knowing which layers you own for each workload, because it differs per compute model within the same organization.
  • Keeping the base image and guest OS patched on every layer you own, which is a continuous obligation and the one most often skipped — see Mutable Servers and Immutable Images.
  • A written debugging order, so that a 03:00 investigation follows the stack rather than the responder's specialism.
How it fails
  • Latency attributed to the application for hours while the cause is CPU throttling against a container limit set six months ago.
  • A process disappearing with no log line and no stack trace: an OOM kill at the isolation layer, visible only in kernel or platform events.
  • A "database problem" that is network-attached disk latency, with every database-internal metric reporting normal.
  • An instance that is slow with idle-looking internal metrics, because another tenant on the physical host is consuming cycles.
  • A change at the fabric layer — a route, a firewall rule, a security group — presenting as a hung connection rather than a refused one, which is why it is misdiagnosed as an application timeout.
How it scales
  • Each layer has its own ceiling and they are hit in a characteristic order: application connections, then container memory, then host file descriptors, then instance network bandwidth.
  • Adding capacity at the wrong layer is the most common wasted intervention — more instances do nothing when the constraint is a per-instance connection limit at the database.
  • The lower the layer, the coarser the scaling unit and the longer the lead time; you add application instances in seconds and physical capacity in weeks.
Security
  • Each boundary is also a trust boundary, and their strengths differ substantially: a hypervisor boundary is far stronger than a container boundary, which is a shared-kernel boundary — see Containers vs Virtual Machines.
  • A vulnerability in a lower layer defeats every control above it, which is why patching the guest OS and base image is a security obligation rather than hygiene.
  • The layers you do not own are also the ones you cannot audit; provider attestations replace your own evidence there — see Shared Responsibility.
  • Multi-tenancy at the physical layer is real. It is well managed by providers and it is not zero risk, which is why regulated workloads sometimes pay for dedicated hosts.
Cost shape
  • You pay at the layer you rent: instance-hours on a VM, request-seconds on a function platform, and node-hours plus a control plane on a managed cluster.
  • Handing a layer to the provider converts operational cost into billed cost — usually a good trade for a small team and a poor one at large steady scale.
  • Fabric layers have their own meters that feel like they should be free: network-attached disk IOPS, cross-zone traffic and storage requests all bill separately from the compute they serve.
What to watch
  • Application: error rate and latency percentiles. Runtime: GC pause time or event-loop lag. Container: throttled periods and OOM events.
  • Host: file descriptors, disk space, connection tracking table occupancy. Instance: steal time and network throughput against the instance ceiling.
  • Fabric: block-device latency and queue depth, which is the metric that exonerates the database.
  • The signal that lies: container CPU utilization during throttling. It reports what the process consumed, not what it was denied, so it looks comfortable while requests queue.
Simpler alternatives
  • A managed platform that removes the bottom four layers from your responsibility entirely. For most teams this is the right trade — fewer layers to instrument, at the price of less visibility when the provider's layer misbehaves.
  • Fewer layers rather than better tools: running a process directly on a VM removes the container and orchestrator layers and removes their failure modes with them. For a single steady service this is a legitimate and underrated design.
  • Bare metal for workloads where the hypervisor leak is intolerable — deterministic-latency systems and some database workloads genuinely benefit, at the cost of everything the virtualization layer was providing.
What adopting this costs
  • Every layer added buys portability, density or automation and charges a new failure mode plus a new place to look at 03:00.
  • Handing layers to a provider reduces your operational surface and reduces your diagnostic reach into exactly the layers you can no longer fix.
  • A thorough layer model costs instrumentation at each level, which is real money and real setup time, and it is the difference between a 20-minute and a 6-hour incident.

The infrastructure layer stack

The infrastructure layer stack
Every running request passes through all seven. Pick a layer to see what it provides upward, how a fault there presents to the layers above, and which domain teaches it properly.
A fault is felt at the layer that reports it, not the layer that caused it. Debugging is walking down.
Provides upward
An isolated filesystem, process namespace and resource envelope — the unit you actually ship and schedule.
Taught in depth by
Operating Systems → namespaces, cgroups, virtualization
How a fault here presents higher up
A process that vanishes with no stack trace (OOM kill), or one that is inexplicably slow while CPU sits at 30% (CPU quota throttling). The application sees a crash it did not cause.
request path, top to bottom:
  application → runtime → container → host → compute → network → physical
  the layer you can observe is rarely the layer that broke
A fault at "Container / VM" never announces itself by that name. It arrives as an application symptom — a process that vanishes with no stack trace (oom kill), or one that is inexplicably slow while cpu sits at 30% (cpu quota throttling) — and the application logs will look innocent, because from the app's point of view nothing wrong happened. The skill this stack teaches is reading a symptom at one layer and forming a hypothesis one or two layers down, then testing that hypothesis with a tool that belongs to that layer rather than to yours.
layer 3 of 7, top to bottom

What people believe, and what is true

Claim

Serverless has no servers, so there is no stack.

Reality

The stack is identical; you just do not own most of it. Cold starts, memory limits and concurrency caps are the same isolation and host layers presenting through a different interface — see Serverless Trade-offs.

Claim

If application metrics look fine, the application is fine.

Reality

Application metrics are measured inside the top layer and cannot see throttling, steal time or fabric latency. They are necessary and nowhere near sufficient.

Claim

Containers isolate like virtual machines.

Reality

They share a kernel. That is a materially weaker boundary and a deliberate trade for density and startup speed — see Containers vs Virtual Machines.

Apply it