Parallelising CI
Wall-clock time is set by the longest dependent chain, not by total work — so parallelism helps exactly as far as the graph and the shared resources allow.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The pipeline is slow and we have runner capacity. What actually gets faster when we parallelise, and what does not?
Total work and elapsed time are different quantities. A pipeline can be mostly idle waiting and still take an hour, and adding machines to a serial chain changes nothing.
Split the test suite across eight runners. Eight times the machines, roughly an eighth of the time.
The serial parts do not shrink. If checkout, dependency install and build take a fixed amount of time before any shard starts, that portion is untouched no matter how many shards there are (Amdahl's Law).
- The serial parts do not shrink. If checkout, dependency install and build take a fixed amount of time before any shard starts, that portion is untouched no matter how many shards there are (Amdahl's Law).
- Shards are only as balanced as the split. Splitting by file count when one file holds the slow integration tests gives you seven idle runners and one long pole.
- Every shard repeats the setup. Eight shards means eight checkouts and eight dependency installs — total compute rises sharply while elapsed time falls modestly.
- Tests that were accidentally serial start colliding: the same database, the same fixed port, the same temporary path, the same third-party sandbox account.
- Concurrency exposes real races that a serial run hid, and they surface as flakes on a change that did not cause them (Flaky Tests).
- And there is a hard ceiling: runner concurrency limits, licence seats, or a shared dependency such as one test database everything contends on.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Elapsed time is the longest path through the job graph — the critical path. Everything off it can be made free by parallelism; nothing on it can (The Critical Path Is the Only Path That Pays).
- That gives two separate optimisations that people conflate. Shortening the critical path is a graph change: remove a dependency, or make a step on the path cheaper. Widening is a capacity change: run more of the off-path work at once.
- Test sharding is a bin-packing problem. Optimal packing needs per-test duration data; without it you are packing by proxy — file count, file size, alphabetical order — and the imbalance is the cost of not measuring.
- Parallelism converts latency into total compute. You are paying more runner minutes for the same work, deliberately (Cost Drivers).
- Hermeticity is the precondition. Two tests can run concurrently only if neither observes the other's state — same process, same filesystem, same database, same external account. Most suites are less hermetic than their authors believe.
Elapsed time is the longest chain
The two pipelines below run the same jobs with the same durations. The first sequences things that have no relationship to each other; the second only makes a job wait when it genuinely consumes another job's output.
The number that changed is not total work — that is identical. It is the longest dependent chain.
install ─▶ build ─▶ unit ─▶ integration ─▶ e2e ─▶ lint every arrow is a wait lint runs last, and lint depends on nothing elapsed = sum of everything
install ─┬─▶ lint
├─▶ typecheck
└─▶ build ─┬─▶ unit ×4 (sharded) ─┐
├─▶ integration ───────┼─▶ report
└─▶ image ─▶ e2e ──────┘
elapsed = install + build + longest branch
total compute: higher. elapsed: much lower.Only jobs that consume another job's output should wait for it. Lint needs the source, not the build. Integration needs the build, not the unit results. Once the graph states that honestly, the runner schedules the parallelism for you — and the remaining duration is a real constraint you can attack rather than an artefact of how the file was written (The CI Dependency Graph).
Splitting a suite
How you split determines the imbalance, and the imbalance determines how much of your parallelism you actually get. All four of these are used in practice and they are not equivalent.
You have four runners and one suite. What decides which test goes where?
when Zero infrastructure, suite roughly uniform. A reasonable starting point.
cost Imbalance is unbounded — one file with the slow tests sets the pipeline duration and the other runners idle.
when You have timing data from previous runs. This is the default answer for a mature suite.
cost Needs somewhere to store timings and a fallback for new tests. Rebalances only as fast as the data refreshes.
when Very long suites with high duration variance; runners pull the next test when free.
cost Needs a coordinator, which is another moving part in CI. Per-test process startup can dominate for fast tests.
when Different tests need genuinely different environments — one shard with a database, one with a browser, one pure unit.
cost Balance is coincidental. But isolation is better, and the failure reports are more legible because each shard means something.
What concurrency breaks
These are the failures that appear the week after sharding is enabled and get blamed on the CI change. Almost all of them are pre-existing defects in test isolation that a serial run was hiding.
The response column matters more than the cause column: serialising makes the symptom go away and leaves the defect, which will reappear the first time production runs the same code concurrently.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Two shards start at once | Random tests fail with unexpected rows or missing records | One shared test database; each shard truncates tables the other is using | One database per shard — a container per job, or a schema namespaced by the shard index |
| Shard starts a local server | Bind failure on a fixed port, or a request answered by the wrong process | Hard-coded port number, shared network namespace on the runner | Bind port 0 and read the assigned port, or isolate each job in its own container (Containers Are Processes With the Kernel’s View Narrowed) |
| Suite run in a different order | A test passes alone and fails in a shard | Order dependence — the test relies on state another test created | Randomise order deliberately and fix what breaks; this defect is real, not an artefact |
| Higher concurrency on the runner | Intermittent timeouts, no clear pattern | CPU or memory contention between shards on the same host, or a genuine race in the code under test | Check runner resource limits first; if the code races under load it will race in production (Reasoning About Races: A Method, Not an Instinct) |
| All shards hit a third party | HTTP 429 from a sandbox or a shared account | Rate limits applied per account, not per runner | Stub the dependency, or serialise only the tests that genuinely need it |
| One shard fails to start | Pipeline reports green | The aggregate result only considered jobs that reported | A fan-in job that asserts the expected shard count, not just the absence of failures |
How to do it properly
Most important first.
- Measure before splitting: find the critical path and the per-step durations. Parallelising off-path work changes nothing you will notice (Measure Before You Optimize).
- Shard by measured duration, not by file count. Most runners can record per-test timings from a previous run and use them to balance.
- Make setup cheap once rather than fast eight times — a warm dependency cache or a prebuilt container image is usually a bigger win than another shard (Caching in CI).
- Give each concurrent job its own database, its own ports and its own temporary directory. Ephemeral containers per shard is the blunt version and it works.
- Set matrix failure behaviour deliberately: cancelling siblings saves compute; not cancelling gives the author every independent failure in one round trip.
- Stop widening when the marginal shard stops moving p95 duration. Past that point you are buying compute and getting queueing.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained within the pipeline — but a wrong verdict from shard interference can wave through a real defect, and then containment is whatever stage catches it next.
What can go wrong
- Shared mutable state between shards — one test database, one Redis, one S3 bucket prefix — producing failures that depend on timing and shard assignment.
- Unbalanced shards where one runs several times longer than the rest, so the pipeline duration is set by a shard nobody looks at.
- Runner starvation: enough concurrent jobs that everything queues, and queue time replaces run time with no net gain (Saturation: The Reading Utilization Cannot Give You).
- Fan-in jobs that need every shard's output — coverage merging, artefact assembly — becoming the new critical path.
- Cancel-on-first-failure hiding independent failures, so the author fixes one, re-runs, and finds the next.
- Cost rising quietly until someone reads the bill, because runner minutes are billed to a budget nobody on the team watches (Cost Awareness).
- "We have N runners so the pipeline is N times faster." Only the parallelisable fraction scales, and the setup you duplicated is not part of it (Why Eight Cores Give You Four and a Half).
- "Flakes appeared after we sharded, so sharding caused them." Sharding revealed them. The race was already there and would eventually have appeared in production, where it is much more expensive (Data Race Is Not Race Condition).
- "Balanced shards means equal test counts." Equal counts, wildly unequal durations. The metric to balance is time.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can name the current critical path and the step on it that dominates.
- Shard durations are within a narrow band of each other, checked rather than assumed.
- Re-running the same commit with a different shard assignment produces the same verdict — evidence that the tests are actually independent.
- p95 pipeline duration moved when you added parallelism. If it did not, the work you parallelised was not on the critical path.
- Runner minutes per merged change is tracked alongside duration, so the trade is visible.
- Reducing shard count is trivially reversible and is the correct first response to shard-interference flakes: collapse to serial, confirm the tests pass, then re-widen while fixing the shared state.
- Collapsing to serial is also the diagnostic. If a failure disappears at one shard, the defect is in the isolation, not in the change under test.
- Automate the split from recorded timings so it rebalances as the suite changes; a hand-maintained shard list is stale within a month.
- Automate per-job resource provisioning — a fresh database container per shard rather than a shared instance with a naming convention.
- Do not automate away the fan-in check. Something must verify that every shard actually reported, or a shard that failed to start reads as a pass.
- Latency bought with compute: the same work on more machines costs more in total, and setup duplication makes the multiplier worse than the shard count.
- Isolation costs resources. A database per shard is the reliable answer and it is also eight databases.
- Duration-based sharding needs historical data, which means the first run after a large suite change is unbalanced.
- High parallelism makes logs harder to read — one failure is now somewhere in eight streams, which raises triage cost unless the reporting is aggregated (Triaging a CI Failure).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGitHub Actions expresses fan-out as
strategy.matrixwithfail-fastcontrolling sibling cancellation; GitLab usesparallel: N(which injects an index into the job) orparallel:matrix; CircleCI has first-class test splitting by recorded timings. The concept transfers, the balancing support does not — some tools give you timing-based splits for free and some make you build it. - SCALE-SPECIFICBelow a few minutes of test time, sharding costs more in setup duplication than it returns. The trade turns favourable when the suite is long enough that duplicated setup is a small fraction of it.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.