Data & Pipeline Parallelism
SIMD and data parallelism against task parallelism, pipeline stages that let items occupy different stages simultaneously, fan-out/fan-in, scatter/gather and the tail-latency cost of waiting for the slowest branch.
A single instruction applies the same operation to four, eight or sixteen adjacent elements at once. It is parallelism with no threads, no locks and no interleavings — and it evaporates the moment the loop body branches per element or the next iteration reads what the last one wrote.
Q · When does my loop turn into vector instructions, and what in the loop body stops that from happening?
Two different shapes of "at the same time". Data parallelism runs one operation over many elements; task parallelism runs different operations at once. They fail differently, share state differently, and scale differently — and most real systems are both at different levels.
Q · Is this the same operation applied to many items, or different operations that happen to be independent — and which one am I actually being offered?
A GPU offers thousands of execution units for work that is uniform, arithmetic-dense and enormous. The reasoning question is almost never "can this run on a GPU" — it is whether the work is big enough and uniform enough to repay moving the data there and back.
Q · Is this work uniform enough, arithmetic-dense enough and large enough to be worth shipping to a separate device and back?
Split the work into stages and let item 4 be read while item 3 is parsed, item 2 is processed and item 1 is written. Throughput rises to the rate of the slowest stage. The time any individual item takes does not improve at all — and usually gets slightly worse.
Q · Can different items occupy different stages at the same time, and what does that actually buy — throughput, latency, or neither?
One incoming request issues N calls concurrently and aggregates the answers. It converts a sum of latencies into a maximum — and simultaneously multiplies your load on everything downstream by N, which is the half nobody plans for.
Q · What does issuing N calls concurrently instead of sequentially buy me, and what does it cost the systems on the other end?
Query every shard in parallel and merge the answers. The result arrives at the speed of the slowest shard, which means the whole request inherits the *tail* of every shard it touched — and querying 100 shards turns a one-in-a-hundred slow response into a two-in-three one.
Q · If every shard is fast 99% of the time, how often is a query across all of them fast — and what do I do about the answer?