The question this answers
When does my loop turn into vector instructions, and what in the loop body stops that from happening?
A loop over a million-element float array multiplying each element by a scale factor and writing it into an output array.
Nothing is shared between actors, because there is only one actor. SIMD runs inside a single instruction stream; the lanes are not threads and cannot observe each other mid-operation.
After the loop, out[i] === in[i] * scale for every i, and no element of out was written more than once — whether the compiler emitted one scalar multiply per iteration or one vector multiply per four.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Four multiplies in the space of one
A scalar mulsd takes one 64-bit float in each of two registers and produces one result. A vector mulpd takes four 64-bit floats packed into each of two wide registers and produces four results — in the same instruction slot, at roughly the same cost. That is the entire idea, and it is the cheapest parallelism on the machine because it buys throughput without buying a scheduler, a synchronization primitive or a single interleaving to reason about.
The compiler does this for you. You write an ordinary loop; the auto-vectorizer notices that iterations are independent, that the memory accesses are contiguous, and that the trip count is known or checkable — and it rewrites the loop to process a *vector width* of elements per iteration, with a scalar "remainder" loop for the leftover elements that do not fill a final vector.
What matters for reasoning about your program is not the register file. It is that vectorization is a property of the loop you wrote, granted at the compiler's discretion, silently withdrawn when the loop stops qualifying. A one-line change inside the body can take a hot loop from four elements per instruction to one, with no error, no warning and no diff in behaviour — only in time.
- The width is a hardware property, not a language one: 2, 4, 8 or 16 elements depending on the instruction set and the element size.
- There is no ordering question between lanes. They are one instruction; nothing can be observed between them.
- The remainder loop is why vectorized code sometimes shows a step change in time at particular input sizes.
in [ a b c d ] <- one 256-bit register, 4x float64
x x x x
in2 [ e f g h ]
--------------------------- mulpd (one instruction)
out [ a*e b*f c*g d*h ]
1,000,000 elements, width 4:
250,000 vector iterations + 0 remainder
1,000,003 elements, width 4:
250,000 vector iterations + 3 scalar remainderWhat stops it: the loop-carried dependency
The vectorizer's precondition is that iteration i does not depend on iteration i-1. A running total, a prefix sum, an in-place shift, a find first match and break — each of these makes iteration i read something iteration i-1 wrote, and four lanes computing simultaneously would all read the *pre-loop* value. The compiler detects this and refuses; when it cannot prove independence (aliased pointers, an opaque function call, an indirect index) it also refuses, because being conservative is the only safe default.
The schedule below is the one the compiler is protecting you from. It is not a thread interleaving — it is what would happen if four lanes of one instruction all read a value that the sequential loop would have updated three times by then. This is worth internalizing precisely because SIMD has no locks: the failure is not a race, it is *the loss of a sequential dependency the algorithm required*.
Branching per element is the other common blocker, and it is more subtle because it does not always block. A branch whose two arms are cheap and side-effect-free can be vectorized by *predication* — compute both arms for all lanes, then blend by mask — which means you pay for both arms in every lane and win anyway if the arms are short. A branch that calls something, allocates, throws, or leaves the loop early cannot be predicated, and the loop goes scalar.
| # | lane 0 (i=1) | lane 1 (i=2) | lane 2 (i=3) | lane 3 (i=4) | State |
|---|---|---|---|---|---|
| 1 | load a[0]=1, a[1]=1 (all four lanes load in the same instruction) | · | · | · | a=[1,1,1,1,1] |
| 2 | · | load a[1]=1, a[2]=1 — a[1] is still the original, not lane 0's result | · | · | a=[1,1,1,1,1] |
| 3 | · | · | load a[2]=1, a[3]=1 | · | a=[1,1,1,1,1] |
| 4 | · | · | · | load a[3]=1, a[4]=1 | a=[1,1,1,1,1] |
| 5 | store a[1] = 1 + 1 = 2 | · | · | · | a=[1,2,1,1,1] |
| 6 | · | store a[2] = 1 + 1 = 2 — should have been 3 | · | · | a=[1,2,2,1,1] ✕ a[2] must be the prefix sum 3, not 2; the lane read a[1] before lane 0 wrote it. |
| 7 | · | · | store a[3] = 1 + 1 = 2 — should have been 4 | · | a=[1,2,2,2,1] |
| 8 | · | · | · | store a[4] = 1 + 1 = 2 — should have been 5 | a=[1,2,2,2,2] |
1// (a) vectorizes: independent, contiguous, no branches2for (size_t i = 0; i < n; ++i) out[i] = in[i] * scale;3 4// (b) usually vectorizes via predication: both arms are cheap and pure.5// Cost: every lane computes both arms, then a mask selects.6for (size_t i = 0; i < n; ++i) out[i] = in[i] > 0 ? in[i] : 0.0;7 8// (c) does not vectorize: the call is opaque, may throw, may alias.9for (size_t i = 0; i < n; ++i) out[i] = expensive_lookup(in[i]);10 11// (d) does not vectorize: loop-carried dependency (see the schedule above).12for (size_t i = 1; i < n; ++i) a[i] = a[i - 1] + a[i];SIMD is a third axis, not a substitute for threads
Engineers routinely file SIMD, multithreading and async under one heading called "making it faster", and then reason badly about all three. They parallelize different things and they compose: a well-written kernel runs on N cores, each core issuing vector instructions, while the process as a whole overlaps I/O asynchronously. Eight cores times four lanes is thirty-two element-operations per cycle-slot, and the two multipliers are independent.
The practical ordering is: vectorize first, then thread. Vectorization costs no synchronization, adds no interleavings, introduces no failure mode you can debug at 3am, and is often the difference between a memory-bound loop and a compute-bound one. Threading a loop that was never vectorized frequently just means N cores are now all waiting on the same memory bus — see Memory Bandwidth: More Cores, Same Bus.
And SIMD does not help waiting. A loop that spends its time in read() gains nothing from wider registers. Classify the work before choosing the axis: Classifying the Work: Computing or Waiting?.
| Axis | What runs at once | Unit | Synchronization | Best for |
|---|---|---|---|---|
| SIMD | Elements within one instruction | Lane | None — one instruction stream | Uniform arithmetic over contiguous arrays |
| Multicore | Independent chunks of work | Thread or task | Locks, atomics, joins | Compute-bound work that partitions cleanly |
| Async I/O | Waiting, not computing | Task | Event loop ordering | Many concurrent waits, little CPU |
| GPU | Thousands of uniform elements | Thread in a warp/wavefront | Barriers within a block | Huge, uniform, arithmetic-dense work |
Key points
- SIMD is parallelism with zero interleavings: the lanes are one instruction, so nothing can be observed between them and no synchronization exists.
- Vectorization is a property of the loop you wrote, granted silently by the compiler and withdrawn silently when the body stops qualifying.
- The blockers are loop-carried dependencies, opaque or side-effecting calls in the body, non-contiguous access, and branches that cannot be predicated.
- A predicated branch is vectorized by computing both arms in every lane and blending — you pay for both arms and still usually win.
- SIMD composes with threads rather than competing: vectorize first, thread second, because vectorization costs no correctness risk.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The compiler analyzes the loop for dependencies between iteration i and iterations before it; any true dependency disqualifies widening.
- • It checks that memory accesses are contiguous and provably non-overlapping, inserting a runtime alias check and two loop versions when it cannot prove it statically.
- • It rewrites the loop to process
widthelements per iteration using packed registers, plus a scalar remainder loop for the tail. - • Branches with cheap, pure arms are converted to predication: both arms execute for all lanes and a mask selects the result per lane.
- • At run time, one packed instruction issues and the lanes complete together; there is no point at which a partial result is architecturally visible.
- • The honest answer: within one vectorized loop there are none. The lanes are a single instruction, so no schedule can interleave them and no other actor can observe a half-completed vector operation.
- • The failure that looks like an interleaving is a loop-carried dependency: all four lanes read
a[i-1]before any lane writes it, so a prefix sum of [1,1,1,1,1] yields [1,2,2,2,2] instead of [1,2,3,4,5]. - • Interleavings reappear the moment you thread the vectorized loop: two threads each running vector code over overlapping ranges of the same array race exactly as scalar code would, and SIMD gives no protection whatsoever.
- • A vectorized read-modify-write of a shared array is not atomic in any useful sense — a wide store is one instruction to the core issuing it and offers no cross-thread guarantee. Use Atomics: What Is Actually Indivisible or Mutexes: What They Protect and What They Do Not for shared elements; see Atomics Are Not Magic.
- • Guarantees: the vectorized loop produces the same results as the scalar loop for integer and bitwise operations, and the compiler will not vectorize when it cannot prove that.
- • Guarantees: no synchronization is required within the loop, because there is no second actor.
- • Does NOT guarantee bit-identical floating-point results if the transformation reassociates additions — that requires fast-math or an explicit reduction rewrite, and it is exactly the effect described in Reduction Ordering: The Sum Changed When the Worker Count Did.
- • Does NOT guarantee vectorization at all. There is no language-level contract; a compiler upgrade, a flag change, or an added
ifcan remove it, and nothing in the type system notices. - • Does NOT make any memory operation atomic, ordered, or visible to another thread. Vector width and memory-model guarantees are unrelated concepts.
- • No lock contention exists here — but a vectorized loop consumes memory bandwidth four to sixteen times faster than the scalar one, which turns a compute-bound loop into a bandwidth-bound one and moves the contention to the memory subsystem.
- • Once every core in the socket is issuing wide loads, the shared last-level cache and the memory controller become the queue. That is the failure in Memory Bandwidth: More Cores, Same Bus, and it is *caused* by successful vectorization.
- • Unaligned or strided access wastes a fraction of every cache line fetched, so the effective bandwidth cost per useful element rises even though the instruction count fell.
- • Silent devectorization: a refactor adds a function call to the loop body, throughput drops by 3-4x, no test fails and no error is logged.
- • Floating-point drift when reassociation is enabled: results change between build configurations, and a regression test comparing exact doubles starts flapping.
- • Wrong results when a programmer hand-vectorizes a loop the compiler correctly refused — the loop-carried dependency case, which produces plausible-looking but wrong output.
- • Aliasing-check overhead: the compiler emits both a vector and a scalar version plus a runtime check, and for short loops the check dominates.
- • Assuming a wide store gives cross-thread atomicity, then writing shared-array code with no synchronization. That is a data race under every language memory model that defines one.
- • Long loops over contiguous numeric arrays: scaling, filtering, dot products, distance calculations, colour conversion, checksums.
- • Work that is already compute-bound with high arithmetic intensity, where more operations per byte loaded is exactly what you want.
- • Hot inner loops you were about to parallelize with threads — vectorizing first often removes the need entirely and costs no correctness risk.
- • Short loops, where the remainder loop plus the alias check outweighs the win.
- • Pointer-chasing and irregular access: linked lists, hash probes, sparse structures. There is nothing contiguous to pack.
- • Loops dominated by unpredictable, expensive branches, where predication makes you pay for the arm you did not want in most lanes.
- • Waiting-bound work, where the CPU is idle anyway. Wider registers do not shorten a network round trip.
- • Ask the compiler: vectorization reports (
-Rpass=loop-vectorize,-Rpass-missed=loop-vectorize,-fopt-info-vec-missed) tell you which loops were widened and, more usefully, the exact reason ones were not. - • Read the disassembly of the hot loop and look for packed mnemonics and register widths — the single unambiguous answer to "did this vectorize?".
- • Compare instructions retired against elements processed. Roughly one instruction per element means scalar; a fraction of that means vector.
- • Watch for a step change in runtime at input sizes just above a multiple of the vector width — that is the remainder loop showing itself.
- • Relying on auto-vectorization is nearly free to write but fragile to maintain: the performance contract is invisible in the source, so document the loops that must stay vectorized and assert on them in a benchmark.
- • Hand-written intrinsics are fast and specific to one instruction set, which means a second scalar implementation, a dispatch path, and twice the tests.
- • Enabling reassociation for a reduction changes numerical results across the whole translation unit unless scoped carefully — a build-flag decision with correctness consequences.
- • Restructuring data from array-of-structs to struct-of-arrays to enable contiguous access is a pervasive change that touches every consumer of that type.
- • Do less work: a better algorithm or an early exit beats a 4x constant factor, and costs no build-configuration risk.
- • A vectorized library routine — BLAS, a SIMD-aware JSON or UTF-8 parser, a numeric array library — which someone has already tuned per instruction set.
- • Thread the loop instead, when elements are independent but each one is expensive; the per-element cost then dwarfs the instruction-issue win.
- • Change the data layout to struct-of-arrays and re-measure before writing a single intrinsic; layout is frequently the whole blocker.
SIMD lanes
What people believe, and what is true
SIMD means my loop runs on multiple cores.
It runs on one core, one thread, one instruction stream. The parallelism is inside a single instruction, and it multiplies with core count rather than replacing it.
A 256-bit store is atomic, so shared arrays are safe.
Width and atomicity are unrelated. A wide store gives no cross-thread ordering or atomicity guarantee under any language memory model — see Data Race Is Not Race Condition.
The compiler vectorizes anything numeric.
It refuses whenever it cannot prove independence and non-aliasing. An opaque call, an indirect index or a running total is enough to disqualify the loop, silently.
Vectorized code always produces identical floating-point results.
Only if the transformation preserves association order. Vectorized reductions typically do not — the sum changes, which is Reduction Ordering: The Sum Changed When the Worker Count Did.
Go deeper
Overview
One instruction, several elements. The cheapest parallelism there is, because there is no second actor and therefore no coordination.
Practical
Write loops the vectorizer can accept: independent iterations, contiguous access, no calls in the body, no early exit. Then check the vectorization report rather than assuming.
Advanced
Restructure data to struct-of-arrays; use predication deliberately for cheap branches; scope reassociation flags to the reductions where you have accepted the numerical consequence.
Internals
Packed registers, masks, alignment, gather/scatter costs and per-microarchitecture instruction throughput are Computer Architecture material — see the bridge below.