Build Performance
Builds are slow for three different reasons — repeated work, serialised work, and genuinely expensive work — and each has a fix that does nothing for the other two.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The build is slow. Which of the three slownesses is it, and what actually helps?
Build duration sets the feedback loop for every change, and it degrades gradually, so by the time anyone measures it the cause is distributed across years of additions.
The build is slow, so add a bigger runner and more parallelism. If that is not enough, cache more aggressively.
A bigger machine does nothing for a serial chain of steps that are waiting rather than computing (Amdahl's Law).
- A bigger machine does nothing for a serial chain of steps that are waiting rather than computing (Amdahl's Law).
- More parallelism does nothing if the graph does not express which work is independent — the tool cannot schedule what it was not told (The CI Dependency Graph).
- More aggressive caching is the dangerous lever: a looser key hits more often and is sometimes wrong, and a wrong build is not a faster build (Caching in CI).
- None of the three helps if the dominant cost is one genuinely expensive step, and without measurement nobody knows which case they are in.
- The fourth response — deleting checks — makes the number better and the signal worse, and it is the one that gets reached for when the first three fail (CI Is a Feedback System).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Build duration decomposes into three quantities that respond to different interventions. Repeated work is doing something that was already done. Serial work is time on the critical path. Expensive work is a single action that genuinely costs what it costs.
- Repeated work is attacked by caching and incrementality, which is a correctness question before it is a performance one (What a Build System Actually Is).
- Serial work is attacked by the graph: remove false dependencies, split large targets so they can overlap, and move work off the critical path (The Critical Path Is the Only Path That Pays).
- Expensive work is attacked by doing less of it — fewer targets, cheaper compiler settings for non-release builds, a different tool — or by moving it off the interactive path entirely.
- The measurement that separates them is per-action timing plus the graph. Without both, "the build takes twelve minutes" is a number with no decomposition and no lever.
- There is also a floor: checkout, dependency restore, container pull and runner startup happen before any of your work. On short builds that overhead can dominate, and no amount of build tuning touches it.
Three slownesses, three levers
Almost every wasted build optimisation is a lever from one row applied to a problem in another. The diagnosis is cheap once you have per-action timings, and impossible without them.
The last column is the one to check yourself against: if the intervention you are about to make matches a wrong fix, stop and measure.
| Kind of slowness | Looks like | Lever that works | Wrong fix that feels right |
|---|---|---|---|
| Repeated work | Rebuilding things that did not change; cache miss rate high | Correct incrementality; content-addressed caching (What a Build System Actually Is) | More runners — you are duplicating the same work faster |
| Serial work | Runners idle; long chain of short actions | Fix the graph; split targets; remove false edges (The CI Dependency Graph) | A bigger machine — the chain is waiting, not computing |
| Expensive work | One action dominates; CPU pinned throughout | Do less: fewer targets, cheaper settings, a different tool | Caching — it is not repeated, so there is nothing to reuse |
| Fixed overhead | Short builds where setup rivals the work | Smaller images, warm runner pools, shallow checkouts | Build tuning — the time is spent before your build starts |
| Contention | Everything slower as parallelism rises | Reduce concurrency to match real capacity (Saturation: The Reading Utilization Cannot Give You) | More parallelism — which is what caused it (Oversubscription) |
Measure the graph, not the feeling
The procedure below is short and almost nobody follows it, which is why so much build optimisation produces no change in duration.
The step people skip is the last one. Optimising a build moves the bottleneck rather than removing it, so a plan made from the first measurement is wrong after the first change.
- 11. Get per-action timings
Turns one duration into a distribution over actions.
fails by Not being available — some build tools emit nothing, and you have to instrument the wrapper.
evidence A list of actions with durations, from a real run rather than a local one.
- 22. Separate overhead from work
Splits runner startup, checkout, container pull and cache restore from the build itself.
fails by Being folded into "build time", so people tune a build whose duration is mostly setup.
evidence Overhead as a share of total, tracked separately for cold and warm runs.
- 33. Find the critical path
Identifies the actions where a local speedup becomes a global one.
fails by Assuming the slowest action is on it. A slow action running in parallel with a slower chain is free (The Critical Path Is the Only Path That Pays).
evidence The longest dependent chain, printed, with each action's contribution.
- 44. Classify
Assigns each expensive action on the path to repeated, serial, expensive or overhead.
fails by Skipping straight to a lever because one is already familiar.
evidence Each action on the path labelled, with the lever named.
- 55. Change one thing and re-measure
Confirms the effect and finds the new bottleneck.
fails by Batching changes, so you cannot tell which one worked — or whether one made it worse.
evidence p95 duration before and after, with the same trigger and the same cache state (Percentiles: Which One, and How Many Users Is That?).
- 66. Verify the cold path
Confirms the speedup did not come from a cache masking a broken clean build.
fails by Never being run, until the release job runs it.
evidence A scheduled cold build that still passes (Reproducible Builds).
No absolute durations appear here on purpose. The decomposition and the ordering transfer between projects; the numbers do not.
The optimisation that is not one
There is one lever that always works on the metric and should be argued separately from all the others, because it does not make the build faster — it makes it check less.
That is sometimes the right call. It is never a performance decision.
before: lint, typecheck, unit, integration, e2e, scan after: lint, typecheck, unit duration: halved coverage: integration, e2e and scan now run nowhere reported as: "build performance improvement" discovered as: an incident, later
before: lint, typecheck, unit, integration, e2e, scan
after: lint+typecheck gate (cheap, first)
unit sharded by measured duration
integration with a warm, correctly keyed cache
e2e moved to trunk, with a fast revert path
scan nightly, with an owner
duration: halved on the PR path
coverage: unchanged; some of it moved and is still enforcedThe first halves the number by deleting the checks that were finding things. The second halves it by ordering, caching and relocating — the same coverage, enforced at a different point, with the trade written down (Designing the Pipeline). If checks genuinely should be removed, remove them as a coverage decision with an owner, not as a side effect of a performance task.
How to do it properly
Most important first.
- Measure first, with per-action timings and the dependency graph together. The critical path is the only place where local speedups become global ones (Measure Before You Optimize).
- Attack repeated work with correct incrementality before attacking anything else — it is usually the largest term and it is bounded by the correctness of your keys.
- Shorten the critical path by removing dependencies that are habits rather than data flow (The CI Dependency Graph).
- Split large targets. A single monolithic compile unit cannot be parallelised or cached partially, so it sets a floor no scheduling can move.
- Measure the fixed overhead separately — startup, checkout, restore — and treat it as its own optimisation problem, because it does not respond to build tuning at all.
- Use different settings for different purposes: debug builds do not need release optimisation levels, and PR builds may not need every target (Designing the Pipeline).
- Re-measure after each change. Optimising a build is bottleneck migration in its purest form; the second bottleneck is rarely where you expected (The Bottleneck Moves After Every Fix).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Slowness itself is contained — it costs time. The dangerous variants are the ones that trade correctness for speed, and those are contained only by a cold build check and by not skipping the checks you removed.
What can go wrong
- Optimising a step that is not on the critical path, producing a real speedup in that step and no change in duration.
- Loosening cache keys for hit rate and importing stale outputs — the one optimisation that trades correctness for speed (Caching in CI).
- Removing checks to hit a duration target, which is a coverage reduction reported as a performance win.
- Adding parallelism past the point where runners contend, so total duration rises while every individual action looks faster (Oversubscription).
- Optimising the warm path only, so the cold build — which is what the release job runs — gets slower unnoticed (Build Environments).
- Measuring the mean when the p95 is what people plan around (Percentiles: Which One, and How Many Users Is That?).
- A remote cache added for speed becoming a hard dependency, so the build fails when the cache service is unavailable rather than falling back to building.
- "The build is CPU-bound, so a bigger machine will fix it." Much of a slow build is waiting — network fetches, container pulls, serialised steps — and none of that responds to more cores (Computing or Waiting?).
- "We doubled the runners and it barely helped." That is the expected result when the critical path is serial. The graph was the constraint, not the capacity.
- "Caching made the build fast." Verify that the cold build still works. A cache that has been hiding a broken clean build is a liability disguised as a speedup.
- "Build time is a developer convenience issue." It sets the feedback loop for every change, which sets change size, which sets incident blast radius (Change Size: Why Small Changes Are Safer, and When They Are Not).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Per-action timings exist and the critical path can be printed. This is the entry requirement; without it every conclusion is a guess.
- The p95 duration moved after a change, not just the p50 or one lucky run.
- Cold and warm build durations are tracked separately, and both are trending in a direction someone chose.
- A scheduled cold build still passes — the speedups did not come from a cache masking an undeclared input (Reproducible Builds).
- The set of checks is unchanged, or its changes were argued explicitly as coverage decisions.
- Build performance changes are configuration and revert cleanly — with the exception of cache key changes, where the previous cache entries may have expired, so the rollback is correct but cold.
- If a speedup came with a correctness risk — looser keys, skipped targets, a wider affected-set filter — revert it and re-measure. A slow correct build is a working build (The CI Dependency Graph).
- When a remote cache is unavailable, the build must degrade to building rather than failing. Verify that fallback deliberately; it is not always the default.
- Automate the collection of build timings so the data exists before anyone needs it.
- Automate a duration regression alarm on the trunk build — build slowness arrives gradually and is never noticed on the day it happens (Regression or Tuesday? Telling a Real Change from Noise).
- Automate cache warming for the paths that matter, with keys strict enough to stay correct.
- Keep the decision to remove a check human and explicit. It is a risk decision, and it should never be a side effect of a performance task (Guardrails, Not Gates).
- Fine-grained targets improve incrementality and parallelism and add per-action overhead; there is an optimum and it is not at either end.
- Aggressive caching buys latency with a correctness risk that scales with how loose the keys are.
- A remote shared cache gives every build warm starts, costs storage and network, and becomes a shared trust boundary (Securing the Pipeline Itself).
- Cheaper build settings for PR builds mean the PR artefact is not the release artefact, which weakens "build once, deploy many" unless the release build is separately verified (Build Once, Deploy Many).
- More runners cost money continuously; the trade against engineer waiting time is real and worth calculating rather than assuming (Cost Awareness).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICWhat is measurable differs sharply: Bazel emits per-action profiles and a critical path summary directly; Gradle has build scans; a shell script offers nothing but timestamps you add yourself. How much of this lesson you can act on depends on which of those you have.
- SCALE-SPECIFICBelow a couple of minutes, fixed overhead — runner startup, checkout, container pull — usually dominates, and build tuning is the wrong target. The three-way decomposition becomes useful once actual build work is the majority of the duration.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — deciding which coverage a faster pipeline may give up, which is the decision the last section refuses to make silently.