I/O Architecture: The Interconnect Is a Shared Resource
Storage, network and accelerators do not each get a private path to memory. They share an interconnect with finite bandwidth, and a saturated link explains slowdowns that look like they belong to whichever device you happened to be watching.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The shape of the tree
A modern machine puts the I/O complex on the CPU package. Devices connect through it, sometimes directly and sometimes through switches that fan a single upstream link out to several downstream ports. On multi-socket machines each socket has its own complex, and a device is therefore *attached to a socket* — reaching it from the other socket crosses the inter-socket link, which is the same effect NUMA: Not All Memory Is Equally Far describes for memory.
This topology is not decoration. It determines which devices share bandwidth, which are one hop from which memory, and which combinations of concurrent activity will interfere. A GPU and a NIC behind the same switch share that switch's upstream link no matter how independent their workloads are.
The practical consequence is that "which slot is it in" is a performance question. On a machine where it matters, moving a card to a different slot can change throughput without changing a line of code — which is deeply unsatisfying and entirely real.
Contention nobody attributed
Each device reports its own metrics against its own capability. A NIC says it is doing four gigabytes per second against a rated ten; an SSD says the same against its own rating. Both look healthy. Neither reports that together they are saturating a shared upstream link, because neither can see it.
This produces one of the more frustrating diagnostic situations in systems work: two subsystems each degrade, each looks fine in isolation, and the cause is a resource that appears in neither set of metrics. The signature is correlation — throughput on one device falls exactly when another device's activity rises, with no code path connecting them.
It matters most for accelerators, because a large host-to-device transfer is one of the few things that can saturate a link on its own. A training or inference job moving data to a GPU can slow down unrelated network and storage traffic on the same branch, which is a genuinely surprising interaction the first time you meet it (see The Transfer You Forgot to Count).
| Observation | What each device reports | What is actually happening |
|---|---|---|
| NIC throughput below rating | Healthy — well under line rate | Sharing an upstream link with a busy accelerator |
| SSD latency up, queue shallow | Healthy — device not saturated | Waiting for interconnect bandwidth, not for media |
| Accelerator transfer slower than expected | Healthy — device idle much of the time | Contending with network traffic on the same branch |
| All three degrade together | Three unrelated problems | One saturated shared link |
Where the bottleneck actually sits
For any I/O path there are several candidate limits: the device's own capability, the link it sits behind, the memory bandwidth the transfer lands in, and the CPU work required per unit of data. The one that binds varies by workload, and only one of them is the device everyone instinctively blames.
The relative scale below is the useful mental model. Reaching a device across an inter-socket hop costs meaningfully more than reaching one attached locally — the same asymmetry NUMA describes for memory, applied to I/O. And a transfer that lands in memory still consumes memory bandwidth, competing with the cores' own demand, which is why heavy I/O can slow down computation that never issues a syscall.
None of these numbers is a measurement of any machine. They are ratios chosen to make the ordering memorable, because the ordering is what transfers and the magnitudes are not.
Key points
- Devices are leaves on a tree, and links in that tree are shared — bandwidth is allocated per link, not per device.
- Two devices can each look healthy in their own metrics while jointly saturating a link neither of them reports.
- On multi-socket machines a device belongs to a socket; reaching it from the other one costs an extra hop.
- Large accelerator transfers are among the few things that can saturate a link alone and slow unrelated traffic.
- The binding constraint may be the device, the link, memory bandwidth or per-byte CPU work — the device is only one candidate.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Core → I/O complex: a request leaves the CPU package through the complex that owns that device's branch.
- 2I/O complex → switch: if the device sits behind a switch, the request shares that switch's upstream link with every sibling device.
- 3Switch → device: the device queues the request against its own capability, which is the only part its metrics describe.
- 4Device → memory: the response is DMAed into RAM, consuming memory bandwidth that the cores are also using.
- 5Cross-socket: if the device belongs to another socket, every step above additionally traverses the inter-socket link.
- • "Each device is below its rated throughput, so I/O is not the problem." Ratings are per device; the shared link has no rating in either device's metrics.
- • "The devices are independent because the code paths are independent." Independence in software says nothing about the topology underneath.
- • "It is a CPU problem because CPU time rose." Per-byte CPU cost rises when data arrives from further away, so the CPU symptom can have an interconnect cause.
- • "Topology is an infrastructure concern." It changes application throughput measurably, and by more than most code-level optimisations would.
Consequences, controls and cost
- • Concurrent I/O on different devices interferes in ways neither device's metrics can explain.
- • The same workload performs differently depending on which slot a card occupies, with no software change.
- • Heavy I/O consumes memory bandwidth and can slow computation that performs no I/O at all.
- • On multi-socket machines, thread placement changes I/O performance as well as memory performance.
- • Place latency-sensitive devices on their own branch rather than behind a switch shared with a bandwidth-heavy one.
- • Pin I/O-heavy threads to the socket that owns the device, so neither the request nor the data crosses sockets.
- • Measure aggregate interconnect utilisation, not just per-device throughput, when several devices are busy together.
- • Where a transfer can be batched or staged, keep large accelerator transfers off the same branch as latency-critical traffic.
- • Aggregate throughput across all devices on a branch, compared against that branch's link capability.
- • Correlate one device's degradation against another device's activity — a tight correlation with no code path between them points at a shared link.
- • Compare the same workload with the process pinned to each socket to expose device locality effects.
- • Watch memory bandwidth utilisation during heavy I/O; DMA traffic competes with the cores for it.
- • Dedicating a branch to a latency-critical device wastes bandwidth that other devices could have used.
- • Pinning threads to a device's socket improves I/O locality and constrains the scheduler, sometimes leaving cores idle.
- • Larger transfers use the link more efficiently and occupy it for longer, delaying anything queued behind them.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICTopology, link widths, switch placement and whether devices are socket-attached vary per machine and per motherboard. Nothing here transfers between systems without checking the actual layout.
- SIMPLIFIEDTreats the interconnect as a bandwidth tree. Real fabrics have per-direction capacity, credit-based flow control and quality-of-service features this model omits.