The question this answers
Is this the same operation applied to many items, or different operations that happen to be independent — and which one am I actually being offered?
An image upload handler that must produce a thumbnail, extract EXIF metadata and run a content classifier (task parallelism) — and, inside the thumbnail step, resample two million pixels (data parallelism).
Data parallelism: a partitioned input array and a partitioned output array, where each worker owns a disjoint index range and shares nothing else. Task parallelism: whatever the different operations happen to touch — typically a result object, a logger, a metrics registry and a connection pool, none of which were designed for it.
Data parallel: every index in the output is written exactly once, by exactly one worker, from the corresponding input index. Task parallel: the aggregate result contains exactly one contribution from each task, and no task's write is lost or partially observed.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The same word for two different shapes
Data parallelism partitions the *input*. There is one operation, applied to a million elements, and the decomposition is arithmetic: give worker k the range [k*n/W, (k+1)*n/W). Every worker runs identical code. Load is balanced by construction if the per-element cost is uniform, and the number of workers is a tuning knob you can turn without changing the program.
Task parallelism partitions the *program*. There are three different things to do, they happen not to depend on each other, and the decomposition is structural: run the thumbnailer, the EXIF reader and the classifier concurrently. Every worker runs different code. Load is whatever it is — the classifier takes 400ms and the EXIF read takes 3ms — and the number of workers is fixed by the number of independent tasks, not chosen.
The distinction matters because it predicts the failure. Data parallel code fails at the partition boundary: overlapping ranges, an off-by-one in the tail chunk, a shared accumulator. Task parallel code fails on *incidental* sharing — the three tasks were written separately by people who assumed they ran alone, and now they both mutate the same result object, or both lazily initialize the same client. See Initialization Races.
| Dimension | Data parallelism | Task parallelism |
|---|---|---|
| What is partitioned | The input, arithmetically | The program, structurally |
| Code per worker | Identical | Different |
| Worker count | A tuning knob — pick any W | Fixed by the number of independent tasks |
| Load balance | Even if per-element cost is uniform; skewed if not | Almost never even — the slowest task sets the join |
| Shared state | Usually none, by construction | Whatever the tasks incidentally touch |
| Typical failure | Overlapping ranges, lost updates to an accumulator | Two tasks mutating a result object; double initialization |
| Scaling ceiling | Memory bandwidth and the serial fraction | The single longest task — see Work and Span |
| Speedup shape | Grows with W until a resource saturates | Caps at total/longest, immediately |
Task parallelism caps immediately; data parallelism has a knob
The upload handler does 3ms + 40ms + 400ms = 443ms of work sequentially. Run all three concurrently and the wall clock is 400ms, because the classifier is still 400ms. That is a 1.1x speedup and it is the *maximum* — no amount of hardware improves it, because the span of the dependency graph is the classifier. This is the single most useful thing to know about task parallelism: its ceiling is set by the longest task and you can compute it before writing any code.
Data parallelism has no such structural cap. Split the classifier's two million pixels across eight workers and the ceiling is set by resources rather than structure — bandwidth, cache, the serial setup — which is a much better place to be, because those can be measured, tuned and bought. See Why Eight Cores Give You Four and a Half.
So the practical move is usually: use task parallelism to overlap what is already there, then look inside the longest task for data parallelism. Parallelizing the 3ms EXIF read is not just useless, it is *negative* — you added a task, a join, an error path and a shared metrics write in exchange for nothing. Parallel Overhead is the honest accounting.
- Task-parallel ceiling: total work / longest task. 443/400 = 1.1x. Compute it first; frequently it ends the discussion.
- Data-parallel ceiling: whatever resource saturates first. Measurable, tunable, sometimes purchasable.
- Parallelizing a 3ms task inside a 400ms request is pure overhead plus a new failure mode.
The task-parallel failure: incidental sharing
Data-parallel workers share nothing because you designed the partition. Task-parallel workers share whatever their code already touched, and nobody designed that — the EXIF reader and the thumbnailer were written a year apart and both call result.addWarning(...) because that was the obvious thing to do when they ran sequentially.
The schedule below is the canonical version: two different tasks doing a read-modify-write on the same result object, and one warning vanishing. Note that neither task contains a loop, a lock or anything that looks concurrent. This is why task parallelism has a higher correctness risk than data parallelism despite offering less speedup — you are not writing new concurrent code, you are *retroactively making existing sequential code concurrent*, and every assumption it made about being alone is now a candidate bug.
The fix is not a mutex around result. It is to give each task its own output and combine after the join — the same discipline data parallelism gets for free. Immutability as a Concurrency Strategy and Copy or Share? are the general form; the specific move here is that every task returns a value instead of mutating one.
| # | EXIF task | Thumbnail task | State |
|---|---|---|---|
| 1 | read result.warnings -> ["orientation missing"] | · | warnings=["orientation missing"] |
| 2 | · | read result.warnings -> ["orientation missing"] | warnings=["orientation missing"] |
| 3 | build ["orientation missing", "exif truncated"] | · | warnings=["orientation missing"] |
| 4 | · | build ["orientation missing", "downscaled past 8x"] | warnings=["orientation missing"] |
| 5 | write result.warnings | · | warnings=["orientation missing", "exif truncated"] |
| 6 | · | write result.warnings | warnings=["orientation missing", "downscaled past 8x"] ✕ The EXIF warning is gone. Both tasks reported success, the response is well-formed, and nothing logged an error. |
Key points
- Data parallelism partitions the input and every worker runs the same code; task parallelism partitions the program and every worker runs different code.
- Task parallelism's speedup ceiling is total work divided by the longest task, and it is knowable before you write anything — usually it is small.
- Data parallelism has a worker-count knob; its ceiling is a resource you can measure rather than a structure you cannot change.
- Data-parallel bugs live at partition boundaries; task-parallel bugs live in incidental sharing that predates the concurrency.
- The productive combination is task parallelism to overlap what exists, then data parallelism inside whichever task turns out to be the span.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Data parallel: choose W, compute disjoint index ranges, dispatch identical closures, join, and (if reducing) combine partial results.
- • Task parallel: identify operations with no data dependency between them, dispatch each as its own task, join on all of them, assemble the result from their return values.
- • The join is the same primitive in both cases (Fork/Join, Promise.all & gather) — what differs is what the workers were asked to do.
- • Load balancing differs fundamentally: data parallel can rebalance by resizing chunks or using Work Stealing; task parallel cannot split a task that was never divisible.
- • Combine step: data parallel usually concatenates or reduces; task parallel usually constructs a record with one field per task.
- • Data parallel, safe: W0 writes out[0..999], W1 writes out[1000..1999], in any order, any number of times interleaved. The invariant holds under every schedule because the write sets are disjoint.
- • Data parallel, broken: an off-by-one gives W0 the range [0..1000] and W1 [1000..1999]. W0 writes out[1000]=a; W1 writes out[1000]=b; the last writer wins and one element is silently wrong exactly once per run.
- • Data parallel, broken by a shared accumulator: W0 reads total (0); W1 reads total (0); W0 writes 500; W1 writes 700 — 500 items vanished from the count while both workers returned success.
- • Task parallel, broken: EXIF reads warnings; Thumbnail reads warnings; EXIF appends and writes; Thumbnail appends and writes — the EXIF warning is overwritten. Neither task contains anything that looks concurrent.
- • Task parallel, broken by double init: both tasks call
getClient(), both find the cache empty, both construct a client, and two connection pools now exist where the code assumed one. See Double-Checked Locking: The Canonical Cautionary Tale.
- • Disjoint index ranges guarantee freedom from write-write conflicts on the output array, with no synchronization at all — the strongest guarantee in this lesson and the reason data parallelism is the safer shape.
- • A join guarantees that every task finished before the combine step reads its result; it establishes happens-before between each task's writes and the joiner's reads. See Happens-Before: The Edge That Makes a Write Visible.
- • A join does NOT guarantee anything about ordering *between* the tasks, so anything that depends on task A running before task B is already broken.
- • Task independence is a claim you assert, not one the runtime checks. Nothing verifies that the classifier does not touch what the thumbnailer writes.
- • Neither shape guarantees the combine step is safe: a reduction over floating-point partials gives a different answer at different W (Reduction Ordering: The Sum Changed When the Worker Count Did).
- • Data parallel: contention is on shared hardware, not shared logic — memory bandwidth, last-level cache, and the odd cache line straddling two workers' ranges (False Sharing: Different Variables, Same Cache Line).
- • Task parallel: contention is on shared objects the tasks did not know they shared — a result record, a metrics registry, a logger, a lazily-initialized client, a connection pool sized for one task at a time.
- • A worker pool shared between the two levels can starve: the classifier submits eight sub-tasks to a pool that the outer three tasks already occupy, and the join waits on work that cannot be scheduled. See Pool Saturation.
- • Lost update on a shared result object between two task-parallel tasks — the schedule above.
- • Overlapping partitions in data-parallel code: a boundary off-by-one produces exactly one wrong element per run, which no unit test with n=10 will catch.
- • Load skew: data-parallel chunks with wildly different per-element costs leave seven workers idle at the barrier while one finishes.
- • Initialization race: two independent tasks each lazily construct the same shared client.
- • Deadlock by pool starvation when nested parallelism submits into the same bounded pool it is running on.
- • Data parallelism helps whenever per-element work is uniform, the elements are independent, and n is large enough to pay for the fork and join.
- • Task parallelism helps most when the tasks are *waiting* rather than computing — three independent HTTP calls overlap almost perfectly, which is Fan-Out / Fan-In: One Request Becomes N.
- • Task parallelism helps when the longest task is not dominant: three roughly-equal 100ms tasks give close to 3x, unlike 3/40/400.
- • Task parallelism over one dominant task: you pay fork, join and error-handling complexity for a 1.1x that a profiler will not even distinguish from noise.
- • Task parallelism over sequential code that was never audited for shared state — the speedup is small and the correctness risk is large, which is the worst trade in this file.
- • Data parallelism on small n, where the fork/join overhead exceeds the work (Parallel Overhead).
- • Data parallelism over skewed elements without work stealing: the barrier waits for the worst chunk, so effective speedup collapses toward 1.
- • Before parallelizing tasks, compute total-work / longest-task. If it is under about 1.3x, stop — the ceiling is not worth the failure modes.
- • For data parallelism, plot speedup against W. A curve that flattens early points at bandwidth or a serial fraction, not at your partitioning.
- • Per-worker completion times at the join: a wide spread means load skew, not insufficient parallelism.
- • For task parallelism, trace spans per task. The critical path is visible directly and tells you which task to look inside next.
- • Task parallelism forces every touched object to become thread-safe or task-local — an audit that spreads well beyond the code you meant to change.
- • It also changes error semantics: three tasks can fail in three different ways at once, and you must decide between first-error, all-errors and partial-result before you can write the join.
- • Data parallelism adds a partitioning function and a combine function, both of which need their own tests, especially at the tail chunk.
- • Nesting the two makes pool sizing genuinely hard: the inner parallelism competes for the same workers as the outer, and the naive answer deadlocks.
- • Do it sequentially and shorten the longest task instead. A 400ms classifier reduced to 150ms beats any task-parallel arrangement of the original.
- • Move the long task out of the request entirely with a background job, when the caller does not need its result now — often the correct answer to the whole problem.
- • For independent I/O, use async concurrency rather than threads: the tasks are waiting, not computing (Async Is Not Parallelism).
- • Vectorize the inner loop (SIMD: One Instruction, Many Elements) before threading it; same shape of win, no interleavings.
CPU parallelism simulator
Amdahl’s term, a synchronisation term, an oversubscription term and a bandwidth ceiling, each one a knob you can switch off. Real curves have more causes than four and are rarely this smooth. There is no ideal core count to read off this chart.
Fork/join and the split threshold
fork(lo, hi): if (hi - lo <= 64) return sequential(lo, hi) // the base case is the tuning knob mid = (lo + hi) / 2 left = spawn fork(lo, mid) // +0.05 ms right = fork(mid, hi) // run one half on THIS thread return left.join() + right // join is where the parallelism ends levels requested 4 → 4 actually taken leaves 16 × 256 elements span 0.91 ms total 1.21 ms
What people believe, and what is true
Running three independent operations concurrently is roughly 3x faster.
Only if they take roughly the same time. 3ms + 40ms + 400ms concurrently is 400ms — 1.1x. The span, not the count, sets the ceiling (Work and Span).
Task parallelism is safe because the tasks are independent.
They are independent in the author's intent. They still share the result object, the logger, the metrics registry and every lazily-initialized singleton they touch.
Data parallelism needs locks around the output array.
Not if the partition is disjoint. Disjoint writes to distinct indices need no synchronization; adding a lock destroys the entire benefit.