Multicoreaffinitypinningschedulinglocalitylatency variance

Thread Affinity: Pinning and Its Price

Affinity constrains which cores a thread may run on. It buys cache locality, NUMA locality and predictable latency, and it costs the scheduler's ability to balance load. It is a genuine tool for latency-critical work and a genuine way to make a machine slower if applied by reflex.

Follow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
When does restricting a thread to specific cores make things faster, and when does it just tie the scheduler's hands?
What you wrote
Threads run somewhere. The scheduler handles it, and which core a thread is on is not something the program thinks about.
What the hardware does
A thread that migrates arrives on a core whose private caches and TLB hold nothing of its working set, and possibly on a different NUMA node from its memory. Affinity prevents that by restricting the set of cores the scheduler may choose.
Affinity is the main lever a programmer has over the hardware-thread mapping. Used on the right workload it removes a real source of tail latency; used indiscriminately it creates idle cores and load imbalance while solving nothing.
Source→Compiler→Instructions→Front End→Execution→Registers→Caches→Memory→I/O→Behavior

What pinning actually buys

Three distinct benefits, worth separating because they apply in different situations. Cache warmth: a thread that stays on one core keeps its L1 and L2 working set and its TLB entries, avoiding the cold-miss burst described in Cache Warmth and the Real Cost of Migration. NUMA locality: a thread pinned to a node keeps accessing memory attached to that node instead of drifting to a socket where all its data is remote. Predictability: a pinned thread does not experience migration at unpredictable moments, which narrows the latency distribution even when it does not improve the mean.

That third benefit is often the real motivation and is worth stating plainly: affinity is frequently a tail-latency tool rather than a throughput tool. Mean throughput may be unchanged or slightly worse while p99 improves noticeably, because the outliers caused by migration disappear.

What affinity buys and costs, by workload
WorkloadBenefit of pinningCost of pinning
Long-running worker, stable working setHigh — keeps caches and TLB warmLow, if the load is even
Latency-critical request handlerHigh — removes migration outliers from the tailIdle capacity when that core has nothing to do
Large parallel job on a multi-socket boxHigh — NUMA locality via node pinningPoor rebalancing if partitions are uneven
Short-lived tasks, bursty arrivalLow — nothing warm to preserveHigh — the scheduler cannot balance the bursts
Mixed workload sharing the machineLow — other tenants disturb the cache anywayHigh — pinned threads cannot move away from noise

What it costs

PLATFORM-SPECIFICAffinity APIs, the numbering of logical CPUs, and whether pinning is permitted at all differ by OS and by container runtime. Inside containers and VMs, the CPU set may be restricted or remapped, so the ids a process sees may not correspond to the physical layout.

The scheduler balances load by moving threads to idle cores. Pinning removes that ability, and the consequences are direct: if a pinned thread's core is busy while another sits idle, the work simply waits, and the machine runs below capacity. Under uneven load, a fully pinned application can be measurably slower than an unpinned one.

There is a second, subtler cost: a pinning layout encodes assumptions about the machine. Core counts, sibling numbering, node boundaries and hybrid core types differ between machines, and a mapping tuned for one can be actively wrong on another — pinning two threads to what turn out to be SMT siblings of one core, or pinning to efficiency cores on a hybrid part. A layout that helps on the development machine can hurt in production.

The honest summary: pin the small number of threads whose latency you actually care about, leave the rest to the scheduler, and re-measure whenever the hardware changes.

  • Pin selectively. A few latency-critical threads, not every thread in the process.
  • Pin to nodes before pinning to cores. NUMA locality is usually the larger effect and is far less brittle.
  • Beware sibling numbering. Logical CPU ids are not always laid out the way you assume; verify the topology.
  • Re-measure on new hardware. A pinning layout is a machine-specific assumption, not a portable optimisation.

The decision, in practice

Affinity is worth reaching for when three things hold at once: the thread is long-lived, it has a working set worth keeping warm, and its latency matters. Miss any of those and the cost usually exceeds the benefit. A short-lived task has no warmth to preserve; a streaming job whose data never fits in cache has nothing to keep; a batch job that only cares about total throughput is better served by letting the scheduler balance.

The corollary is that affinity is a poor *first* response to a performance problem. If parallel code is not scaling, the likelier causes are the ones in this module — write sharing, false sharing, bandwidth saturation — and pinning does not fix any of them. Measure the migration rate first: if threads are not actually migrating much, affinity has nothing to offer.

Reflex pinning: every thread pinned
1// One thread pinned to each logical CPU,
2// on a machine shared with other work.
3for (i = 0; i < num_logical_cpus; i++)
4    pin(worker[i], cpu = i);
5 
6// Uneven task lengths now cannot rebalance:
7// worker 3 has a queue, worker 7 is idle,
8// and the scheduler is not allowed to help.
9// Some pairs are SMT siblings of one core.
Selective pinning: only what needs it
1// Pin only the latency-critical thread, and
2// pin it to a node rather than a single core.
3pin_to_node(latency_critical_thread, node = 0);
4 
5// Everything else stays schedulable, so the
6// OS can still balance bursts and uneven work.
7// The tail improves where it matters; the rest
8// of the machine keeps its flexibility.

Affinity trades scheduler flexibility for locality. Spending that trade on every thread pays the cost everywhere and collects the benefit only where warmth actually existed. Spending it on the few threads whose tail latency matters collects most of the benefit for a fraction of the cost.

Key points

  • Affinity buys cache warmth, NUMA locality and predictability by restricting where a thread may run.
  • It is often a tail-latency tool rather than a throughput tool — p99 improves while the mean may not.
  • The cost is scheduler flexibility: pinned threads cannot move to idle cores, so uneven load leaves capacity unused.
  • Pinning layouts encode machine-specific assumptions and can be actively wrong on different hardware.
  • Pin selectively, prefer node-level pinning, and verify threads are actually migrating before reaching for it.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    Process → OS: an affinity mask restricts the set of logical CPUs the scheduler may place this thread on.
  2. 2
    Scheduler → placement: on each dispatch, only cores in the mask are considered, so the thread returns to the same caches.
  3. 3
    Core → private caches: the working set and TLB entries survive between slices instead of being rebuilt.
  4. 4
    Thread → NUMA node: pinning to a node keeps accesses on the local memory controller rather than crossing the link.
  5. 5
    Load imbalance → idle cores: work queued behind a pinned thread cannot migrate, so capacity elsewhere goes unused.
What people conclude from this — wrongly
  • • "Pinning always helps" — it helps long-lived threads with warm working sets, and hurts bursty or uneven workloads.
  • • "Pin every thread for maximum performance" — that pays the flexibility cost everywhere and collects the benefit rarely.
  • • "Affinity will fix my scaling problem" — write sharing and bandwidth saturation are far likelier causes and are unaffected.
  • • "CPU 0 and CPU 1 are two cores" — they may be SMT siblings of one core; verify before assuming.

Consequences, controls and cost

What it causes
  • • Narrower latency distributions for pinned latency-critical threads.
  • • Reduced throughput under uneven load, because the scheduler cannot rebalance.
  • • Pathological placements when logical CPU numbering differs from the assumed layout.
  • • Behaviour that changes between machines, and between bare metal and containers.
What you can do
  • • Pin only the threads whose latency you care about, and leave the rest schedulable.
  • • Prefer NUMA-node affinity over single-core pinning: most of the benefit, far less brittleness.
  • • Verify the topology and logical CPU numbering programmatically rather than hard-coding a layout.
  • • Measure migration rate before pinning; if threads are not migrating, affinity has nothing to offer.
How to see it
  • • Measure the thread migration rate first; low migration means affinity has little to offer.
  • • Compare p50 and p99 pinned versus unpinned — the tail is where the benefit usually shows.
  • • Watch for idle cores alongside queued work, which is the signature of over-pinning.
  • • Verify the actual topology and sibling mapping rather than assuming logical CPU numbering.
What it costs
  • • Locality gained is scheduling flexibility lost; under imbalance that costs throughput.
  • • A pinning layout is machine-specific and can degrade or misfire on different hardware.
  • • Node pinning is more robust than core pinning but gives up the last increment of cache warmth.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • PLATFORM-SPECIFICAffinity interfaces, CPU numbering and whether pinning is allowed differ by OS, container runtime and hypervisor. Inside containers the visible CPU set may be restricted or renumbered relative to the host.
  • MICROARCH-SPECIFICOn hybrid CPUs with performance and efficiency cores, a pinning layout that ignores core type can pin latency-critical work onto efficiency cores and make things worse.

Misconceptions

Claim
“Pinning threads to cores always improves performance.”
Reality
It improves locality and predictability for long-lived threads with warm working sets. For bursty or unevenly-sized work it removes the scheduler's ability to balance and can reduce throughput outright.
Claim
“Affinity is how you get more CPU for your process.”
Reality
It constrains placement; it does not grant priority or capacity. A pinned thread on a busy core gets less CPU than an unpinned thread free to move to an idle one.
Claim
“Logical CPU numbers map straightforwardly onto physical layout.”
Reality
Numbering schemes vary: adjacent ids may be SMT siblings of one core, or spread across sockets. Pinning two threads to "different CPUs" can land them on one core. Always read the topology.

Apply it

🔍 Scales to Sixteen Threads, Falls Over at Sixty-Four

Where the rest of this lives

Concurrency & Parallelism
Work stealing and load balancing

Work-stealing schedulers deliberately move tasks between workers to balance load, which is in direct tension with affinity. Choosing between locality and balance is a concurrency design decision; the locality half of the trade is quantified here.

← NUMA: Not All Memory Is Equally FarCache Warmth and the Real Cost of Migration →