The question this answers
My parallel job is fast on a single-socket machine and slow on a bigger dual-socket one — why would more hardware be worse?
A parallel scoring pass over a 200 GB in-memory dataset, on a two-socket machine, where the main thread allocates and populates the array before the workers start.
The dataset pages — shared in the sense that any worker may read any page, and each page physically lives in exactly one socket's memory. Nothing is mutated concurrently; the sharing that costs you is *physical placement*, not logical state.
Every worker reads the same values regardless of which socket holds the page. Placement changes the latency of an access, never its result — this is a performance property with no correctness component at all.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Two sockets, two memory pools, one address space
A multi-socket server presents one flat virtual address space, and beneath it the memory is physically divided: each socket has its own memory controllers and its own attached DRAM. An access to memory attached to your own socket is *local*. An access to the other socket's memory is *remote*: it traverses the inter-socket link, and it costs more in latency and consumes a shared interconnect that has far less bandwidth than either socket's local path to its own DRAM.
Nothing in your program mentions any of this. Pointers do not say which socket their pages live on, allocation APIs do not ask, and the code is identical on a one-socket and a two-socket machine. That is what makes NUMA surprising: it is the only performance property in this module that gets *worse* when you move to a bigger machine, and the reason is that the bigger machine is not one machine with more cores — it is closer to two machines that can read each other's memory at a discount.
The practical consequence is a rule, not a mechanism: a worker should read memory that its own socket owns. Everything else in this lesson is about how pages end up on the wrong socket and what to do about it. The interconnect protocol, the coherence directory and the actual latency ratios are hardware, and belong to Computer Architecture.
- One virtual address space, several physical memory pools. Nothing in the language exposes the difference.
- Remote access is not an error and not a fallback — it just costs more, on every access, forever.
- The interconnect is shared by every remote access on the machine, so a badly-placed job also slows down its neighbours.
How pages end up on the wrong socket: first touch
The common default placement policy is *first touch*: a page is allocated on the memory node of whichever thread first writes to it, not whichever thread allocates the virtual range. That policy is a good default — it puts memory near the thread that is actually using it — and it becomes a trap in the single most common initialization pattern in parallel programs: one thread allocates and populates the whole dataset, then N workers process it.
The schedule below is that trap. The main thread runs on socket 0 and touches all 200 GB, so every page lands on memory node 0. The sixteen workers on socket 1 then do half the reads, and every one of those reads is remote. The job did not do anything wrong that a code reviewer would notice; it initialized its data and then parallelized its loop. On a single-socket machine this code is optimal, and it stays optimal-looking on the bigger machine while running measurably slower.
The fix follows directly from the mechanism: initialize in parallel, with the same partitioning you will use to process. If worker k writes the pages it will later read, first touch places them on worker k's node and everything is local. This is the standard NUMA remedy and it usually costs nothing but moving the initialization loop inside the same parallel construct as the processing loop. Where that is impossible — the data arrives from a file, or from another process — the alternatives are an interleave policy (spread pages across nodes so nobody is systematically penalized and the interconnect load is balanced), or explicit binding of workers and their data to the same node.
| # | Main thread (socket 0) | Worker 3 (socket 0) | Worker 19 (socket 1) | State |
|---|---|---|---|---|
| 1 | allocate 200 GB virtual range | · | · | node 0 pages=0 node 1 pages=0 |
| 2 | populate the array: first write to every page | · | · | node 0 pages=200 GB node 1 pages=0 |
| 3 | start 32 workers, 16 per socket | · | · | node 0 pages=200 GB node 1 pages=0 |
| 4 | · | read its slice — pages are on node 0, same socket | · | w3 accesses=local |
| 5 | · | · | read its slice — pages are on node 0, other socket | w19 accesses=remote ✕ Half the workers read entirely remote memory, over a shared interconnect, for the whole run. |
| 6 | · | · | continues; every access crosses the link | interconnect=saturated by 16 workers w19 throughput=well below w3 |
Placement policies, and when to care at all
The honest framing for most engineers is: you probably do not have this problem, and you should check before spending an hour on it. A container pinned to eight cores of one socket is not a NUMA workload. A JVM on a single-socket cloud instance is not a NUMA workload. NUMA effects appear on large multi-socket servers running memory-heavy parallel jobs — in-memory databases, analytics engines, large caches, HPC — and the first question is always whether the machine has more than one node at all.
When it does, the policies below are the vocabulary. Interleave is the low-effort, low-risk option: it does not make anything local, it makes the penalty uniform and spreads interconnect load, which is a good default for a shared cache or any structure accessed from everywhere. Bind is the high-effort, high-reward option: pin workers to a node and place their data on the same node, which is how in-memory databases get their numbers, and which brings all the portability and flexibility costs of Thread Affinity: Pinning, and What It Costs You along with it.
And note the interaction with everything else in this module. NUMA multiplies the Memory Bandwidth: More Cores, Same Bus problem — remote accesses consume the interconnect as well as DRAM bandwidth — and it makes Parallelism Can Destroy Locality more expensive, because a migration that also crosses sockets turns every subsequent access remote until the pages are moved or the thread migrates back. Migration across a socket boundary is the most expensive scheduling decision on the machine, which is precisely why schedulers try to avoid it and why the topology is worth knowing before you tune anything.
- Check the topology first. One node means none of this applies and the time is better spent elsewhere.
- Parallel first touch is the highest ratio of benefit to effort in this lesson, by a wide margin.
- Cross-socket thread migration is the most expensive scheduling event on the machine, because it makes an entire warm working set remote.
| Policy | What it does | Effort | Good for | Risk |
|---|---|---|---|---|
| Default (first touch) | Page lands on the node of the first thread to write it | None | Programs whose init is already parallel | Serial init places everything on one node — the trap |
| Parallel first touch | Initialize with the same partitioning you process with | Low — move one loop | Any parallel-for over a large array | Init and processing partitioning must stay in sync |
| Interleave | Round-robin pages across nodes | Low — a policy flag | Structures read from everywhere: caches, shared indexes | Nothing is local; you chose uniform mediocrity |
| Bind workers + memory to a node | Pin threads and allocate on the same node | High | In-memory databases, HPC, latency-critical services | Loses scheduler flexibility; can idle a whole socket |
| Shard the process per node | Run one process per socket, partition the data | High | Large caches and stores that can shard cleanly | A distributed system inside one machine |
Key points
- On a multi-socket machine, memory is several pools; local access is fast and remote access crosses a shared interconnect at a real cost.
- Placement is decided by first touch — the thread that first writes a page, not the one that allocated it.
- The classic trap is serial initialization followed by parallel processing: every page lands on one node and half the workers read remotely forever.
- The classic fix is to initialize with the same parallel partitioning used to process, which is usually a few lines.
- Check whether the machine has more than one node before doing any of this; on a single-socket instance or a pinned container it does not apply.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Each socket has its own memory controllers and attached DRAM, exposed as a NUMA node; all nodes appear in one virtual address space.
- • An allocation reserves virtual addresses without placing physical pages.
- • On the first write to a page, the OS allocates a physical page — under the default policy, on the node of the writing thread.
- • Subsequent accesses from that node are local; accesses from another socket traverse the interconnect, adding latency and consuming a shared, comparatively narrow link.
- • The page stays where it was placed unless the OS migrates it (some kernels do this automatically, with their own overhead) or the program asks.
- • The trap: main thread touches every page (all on node 0); workers on both sockets start; socket-1 workers read remotely for the entire run while socket-0 workers read locally. No incorrect result, a large and permanent cost asymmetry.
- • The fix: worker k first-touches its own slice; worker k later reads its own slice — every access local, on both sockets, with no policy flags or pinning.
- • The migration case: worker 7 runs warm on socket 0 with local pages; the scheduler migrates it to socket 1; every subsequent access to its working set is now remote until it migrates back or the pages are moved.
- • The shared-structure case: all 32 workers read one lookup table that lives on node 0. Sixteen of them are remote no matter what you do — this is where interleave, or replicating the table per node, earns its keep.
- • The allocator case: a pooled allocator hands worker 19 a buffer whose pages were first touched by worker 3 on the other socket, so a freshly "allocated" buffer is remote from birth.
- • Guarantees correctness regardless of placement: a load returns the same value whether the page is local or remote, and coherence is maintained across sockets.
- • First touch guarantees a page is placed on the node of the thread that first wrote it, under the default policy — which is exactly why serial initialization is a placement decision.
- • Nothing guarantees a thread stays on the socket it started on, so locality established at initialization can be lost by a scheduling decision.
- • Nothing guarantees pages migrate to follow a thread. Some kernels attempt it; it is a heuristic with its own cost, not a promise.
- • An interleave policy guarantees a uniform penalty, not a low one — it removes the worst case and removes the best case with it.
- • The interconnect is a shared resource for every remote access on the machine, so a badly-placed job degrades unrelated processes.
- • Node 0's memory controllers are contended by all 32 workers in the trap case, while node 1's sit idle — half the machine's memory bandwidth is unused.
- • Shared read-only structures accessed from every socket concentrate traffic on one node; replicating them per node trades memory for interconnect relief.
- • Cross-socket migration adds contention indirectly: the migrated thread's working set becomes remote, so its bandwidth demand moves onto the link.
- • A job that is slower on a larger multi-socket machine than on a smaller single-socket one, with no code change and no obvious cause.
- • Half the workers systematically slower than the other half, in a job with perfectly symmetric work.
- • Throughput variance run to run, depending on which socket the main thread happened to start on.
- • A pooled or arena allocator handing out remote memory because the pool was populated by a thread on another socket.
- • Automatic page migration heuristics thrashing under a fluctuating access pattern, adding cost without settling.
- • Aggressive node binding that idles an entire socket when load is uneven — the fix becoming the problem.
- • Large in-memory workloads on multi-socket servers: databases, caches, analytics engines, anything holding tens or hundreds of gigabytes resident.
- • Parallel batch jobs whose initialization is separate from their processing, where the parallel-first-touch change is small and the win is large.
- • Latency-critical services on dedicated multi-socket hardware, where binding workers and their data to one node removes a whole class of variance.
- • On single-socket machines, cloud instances with one node, or containers pinned within a socket — the concept simply does not apply and the tuning is wasted effort.
- • When binding is applied without measuring: you have traded scheduler flexibility, portability and the ability to use an idle socket for a benefit you did not confirm.
- • When the working set is small enough to live in cache, where placement rarely matters.
- • When it distracts from a larger problem: a job that is bandwidth-bound on one socket will still be bandwidth-bound on two, and NUMA tuning will not fix it.
- • First: how many NUMA nodes does the machine have? On a one-node machine the answer to every question in this lesson is "not applicable".
- • Remote access ratio — the fraction of memory accesses served by a non-local node — which is the direct measure, where hardware counters are available.
- • Per-worker throughput grouped by socket. A systematic split between two halves of your workers is the signature, and it needs no special tooling.
- • The A/B that needs no counters: run the job confined to a single socket. If per-core throughput improves, placement is your problem.
- • Interconnect utilization, when exposed. A saturated link explains why adding workers on the second socket stopped helping.
- • Topology awareness leaks into application code: initialization order, allocator choice and thread placement all become things you must reason about together.
- • Init and processing partitioning must stay consistent, and nothing enforces that — a later refactor of one loop silently undoes the placement.
- • Binding requires knowing the deployment topology, which conflicts with running the same binary on heterogeneous machines and inside schedulers that move you.
- • Sharding a process per node turns one program into several coordinating processes, with all the routing and failure-handling that implies.
- • It is very hard to test: the effect does not reproduce on a developer laptop or a single-node CI runner at all.
- • Confine the job to one socket. Half the cores, all local memory, and frequently faster than using the whole machine badly — and it is a launch-time flag, not a code change.
- • Run one process per socket over a partitioned dataset, letting the operating system's default placement do the right thing within each.
- • Use a smaller single-socket machine, when the workload fits. Fewer NUMA effects, easier reasoning, often better price/performance.
- • Reduce the working set so it fits in cache, which makes placement much less important.
- • Set an interleave policy and move on: a small, uniform penalty with a one-line change is often the right amount of effort for this problem.
What people believe, and what is true
RAM is RAM — an address is an address.
One address space, several physical pools. Which pool a page is in decides how far every access to it has to travel, for the lifetime of that page.
The page is placed where I allocated it.
Allocation reserves addresses. Placement happens on first write, on the node of whichever thread does that write — commonly the main thread, for the whole dataset.
A bigger machine is always at least as fast.
A dual-socket machine running a serially-initialized parallel job can be slower than a single-socket one, because half the workers now read across the interconnect.
NUMA tuning is something every parallel program needs.
Most workloads run on one node and are unaffected. Check the topology first; on a single-node machine there is nothing here to tune.