SIMD: One Instruction, Many Elements

A vector register of a given width, divided by the element size, gives the number of lanes. The same physical register holds a different number of elements depending on the type: twice as many 32-bit floats as 64-bit doubles, four times as many 8-bit integers as 32-bit ones. Narrower types therefore vectorise better, which is one of the real performance arguments for using the smallest type that is correct.

Each lane is independent. Lane 3 computing a[3] + b[3] cannot observe or affect lane 0, and there is no communication between them in a basic arithmetic operation. That independence is exactly why the requirement on your code is that the iterations be independent — the hardware has no mechanism to carry a value from one lane to the next.

Operations that need to cross lanes — a horizontal sum, a shuffle, a reduction — exist but are a different and generally more expensive class. This is why reductions vectorise into per-lane partial results followed by one horizontal combine at the end, exactly mirroring the multiple-accumulator technique from Dependency Graphs: The Real Shape of Your Code.

Three conditions, and all three must hold. Independence: iteration i must not depend on iteration i-1, because lanes execute together and cannot be ordered relative to one another. Contiguity: the elements should be adjacent in memory, so one wide load fills a register — gathering scattered elements is supported on some instruction sets but costs far more. Uniformity: every lane does the same operation, so data-dependent branching within the loop body forces the hardware to execute both paths and discard the unwanted lanes.

The loop below satisfies all three and is the canonical vectorisable shape: read contiguous, compute uniformly, write contiguous, no cross-iteration dependency. The scalar version does the same arithmetic one element at a time; the vector version does it several elements per instruction, with the same total number of additions performed by fewer instructions.

Notice the trip-count condition hiding at the end. Vector loops need a scalar remainder for elements that do not fill a final register, and the setup has a fixed cost. For very short loops that overhead can exceed the benefit, which is one legitimate reason a compiler declines to vectorise something that looks eligible.

Vectorisation increases the rate at which a core consumes memory. A loop doing one add per element loaded is already close to memory-bound in scalar form on many machines; making the arithmetic eight times faster does not help when the limit was fetching the data. The metric that predicts this is arithmetic intensity — operations performed per byte moved — and low-intensity kernels see little benefit (When the Memory Bus Is the Bottleneck).

There is also a frequency consideration on some designs. Sustained heavy vector work can draw enough power that the core reduces its clock, so the vector speedup is partially offset by every instruction running at a lower frequency, including the scalar code around it. This is strongly MICROARCH-SPECIFIC — it is pronounced on some server parts with wide vector units and negligible elsewhere — and it is a real reason to measure rather than assume (The Clock Is a Variable, Performance Per Watt).

And branches inside the loop body undo the uniformity requirement. When lanes need to take different paths, the usual implementation executes both sides and merges with a mask, so the loop pays for all paths regardless of the data. A branchy loop can be slower vectorised than scalar, which connects directly to Branchless Code: A Trade, Not an Upgrade and to why predictability matters even where there is no branch predictor involved.