CI Is a Feedback System
The product of a pipeline is a trustworthy verdict delivered while the author still has the change in their head; everything else is overhead.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is CI actually for, given that the tests would run eventually anyway?
A defect costs more to correct the further it is discovered from the decision that created it — not because the fix is bigger, but because the context needed to make it is gone.
CI is where the tests run. Its job is to be thorough: run everything, on everything, so nothing gets through. How long it takes is a secondary concern.
A pipeline that takes longer than an engineer's attention span converts a correction into an interruption. The author has already opened the next change; coming back costs a full context reload.
- A pipeline that takes longer than an engineer's attention span converts a correction into an interruption. The author has already opened the next change; coming back costs a full context reload.
- Long pipelines push people to batch. If each verification costs half an hour of waiting, you stop pushing five small commits and start pushing one big one — which makes every verdict ambiguous (Change Size: Why Small Changes Are Safer, and When They Are Not).
- People route around slow feedback. They merge on a partial signal, they skip the local check because CI will catch it, or they add
[skip ci]— each of which moves discovery further from the decision. - Thoroughness with no ordering means the two-second lint failure is reported after the twenty-minute end-to-end suite has also failed, for the same reason.
- And a pipeline that is wrong sometimes is worse than a slow one: an untrusted verdict is not a signal at all, it is a coin flip people learn to re-flip (Flaky Tests).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- CI is a control loop. It has a measurement (the checks), a latency (how long until the verdict arrives), and a trust level (how often the verdict is right). Its value is the product of all three, and any one at zero makes the others irrelevant.
- Correction cost is dominated by context loss, not by code size. The same one-line fix is trivial in the editor, cheap in review, awkward after merge, and an incident in production — because at each step more people, more state and more coordination are involved.
- That gives the design rule: push each class of defect to the earliest point that can detect it. Not the earliest point that *could* run the check — the earliest that can produce a correct verdict for that class.
- Latency matters in bands rather than linearly. Under about a minute the author waits; under about ten minutes they stay on the change; beyond that they context-switch, and the correction becomes a scheduled task instead of a continuation.
- Trust is the multiplier. A check that is red 5% of the time for reasons unrelated to the change stops being read; a check nobody reads has zero latency value no matter how fast it is.
Correction cost is context cost
The same defect is not the same problem at different points in the loop. What changes is not the size of the fix — it is how much has to be reconstructed before anyone can make it, and how many people are involved.
Read the last column as the real currency. Everything else on the row is a consequence of it.
| Discovered at | Who is looking | Context still loaded | What correcting costs |
|---|---|---|---|
| Editor / language server | The author, mid-thought | Everything | Keystrokes |
| Pre-commit hook | The author, about to commit | Everything | Seconds; no one else involved |
| Fast PR check | The author, still on this change | Most of it | A push; the review has not started |
| Slow PR check | The author, now on the next change | Partial | A context switch back |
| Post-merge on trunk | Whoever notices | Little; possibly not the author | Revert, notify, re-do (Continuous Integration) |
| Staging | A tester or nobody | None | Reproduce, bisect, re-run the whole path |
| Production canary | An operator | None | Roll back, investigate under pressure (Canary Analysis: Compared Against What?) |
| Production, all traffic | Customers | None | An incident, with coordination cost dominating (What Happens Between the Page and the Postmortem) |
Two outputs: a verdict and a delay
Every pipeline produces both. Teams instrument the delay because it is easy to measure, and rarely instrument the verdict quality because it is not — which is how a pipeline gets faster and less useful at the same time.
The compare below is two pipelines with identical checks. The difference is entirely in ordering and in what the failure tells you.
push
└─ one job, everything in sequence
install → build → unit → integration → e2e → lint
fails at e2e because of a typo lint would have caught
reports: "Process completed with exit code 1"
author: opens UI, scrolls 4000 log linespush
├─ lint + typecheck (fast, cheap, high signal)
│ └─ fails → stop here, report the file and line
└─ build → unit (sharded) → integration
└─ e2e only after unit is green
reports: file, line, expected vs actual, seed, artefact linkBoth run the same checks and would reach the same verdict. The first delivers it after the most expensive stage and in a form that requires archaeology; the second delivers it in the first minute in a form the author can act on. Same coverage, different product.
The attention window
Duration targets are usually stated as round numbers with no reasoning attached. The reasoning that matters is behavioural: at each band, the author does something different, and what they do determines whether the verdict lands as a correction or as a ticket.
This is why a pipeline that gets 20% faster can be worth nothing and one that crosses a band boundary can change how a team works.
- 1Under ~1 minute
The author watches it. Failure is a continuation of the same edit.
fails by Almost nothing fits here except lint, typecheck and fast unit tests — trying to fit more means skipping checks.
evidence Fix commits follow the failure within minutes, by the same person.
- 2~1 to ~10 minutes
The author stays on the change — reads the diff again, writes the PR description.
fails by Variance. A p50 of six minutes with a p95 of twenty-five behaves like the p95.
evidence p95 duration, not mean, tracked over time (Percentiles: Which One, and How Many Users Is That?).
- 3~10 to ~40 minutes
Context switch. The change becomes something to come back to.
fails by Correction cost jumps discontinuously here, and batching begins.
evidence Rising average commits per push; rising time from failure to fix.
- 4Beyond that
The pipeline becomes a scheduled event, not feedback.
fails by People merge on partial signal and rely on post-merge or staging to catch things.
evidence Merges happening before all checks report; a growing share of defects caught after merge.
- 5Any duration, untrusted
The author re-runs the job.
fails by Trust is the multiplier — at zero trust, every other number is irrelevant.
evidence Re-run rate per pipeline, and the share of re-runs that pass unchanged (Flaky Tests).
The band boundaries are behavioural rules of thumb, not measurements — the point is that the cost is a step function, not that it steps at exactly these numbers.
How to do it properly
Most important first.
- Decide, per class of defect, where it should be caught — and make sure something there actually catches it. Formatting belongs in the editor, not in a pipeline job.
- Order checks so the cheapest high-signal ones report first, and let their failure short-circuit the expensive ones (Designing the Pipeline).
- Treat pipeline duration as a product metric with an owner, measured at the percentile people actually experience — the slow runs are the ones that change behaviour (Percentiles: Which One, and How Many Users Is That?).
- Protect trust aggressively: quarantine flaky checks rather than letting them dilute every verdict (Flaky Tests).
- Report failures in a form the author can act on without opening the pipeline UI — the assertion, the diff, the seed, the artefact (Triaging a CI Failure).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A single bad verdict is contained by review and by later stages — but eroded trust is not contained by anything, and shows up as changes shipping on signals nobody reads.
What can go wrong
- Optimising duration by deleting checks, which improves the metric and degrades the signal — the one CI change that is strictly negative and always looks like progress.
- Optimising the median while the p95 run is what people plan around.
- Adding a check nobody owns, which goes flaky, gets marked advisory, and now consumes runner time to produce a result nobody reads.
- Fast feedback on the wrong thing: a two-minute pipeline that only lints, giving confident green verdicts about a class of defect it never examines.
- Notification design that buries the verdict — an email nobody opens, a channel with hundreds of messages a day (Alert Fatigue).
- "Faster CI means less testing." It usually means the same testing, ordered and parallelised differently. Deleting checks is a coverage decision, and should be argued as one.
- "CI is where quality happens." CI reports on quality. It is a measurement device; pointing a faster instrument at the same code does not change the code.
- "If it is green we can ship." Green is a statement about the artefact. Whether to ship is a statement about rollout risk, and CI knows nothing about that (A Successful Deploy Is Not Evidence of a Healthy System).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Time from push to first actionable failure, at p95, is measured and trending in a direction someone chose.
- The share of pipeline runs that failed for a reason unrelated to the change is measured, and it is small.
- When a check fails, the failure text alone is enough to start fixing — verified by asking someone who did not write the check.
- Defects are being caught at the step you designed them to be caught at, not systematically one step later.
- Adding a required check is reversible: demote it to advisory, which keeps the data while removing the block. Removing a check is much harder to reverse, because the coverage gap is invisible until something exploits it.
- If a newly added check turns out to be flaky, quarantine it the same day. Leaving it required "for a week to see" is how a team learns to ignore red.
- Automate the loop mechanically: trigger, run, report, annotate the change with the verdict.
- Automate flake detection — repeated pass/fail on identical input is machine-detectable and humans are bad at noticing it.
- Do not automate "re-run until green". Automatic blanket retries convert a real intermittent defect into invisible latency, which is the exact failure this lesson exists to prevent (The Automation Trap).
- Fast feedback usually costs parallel compute — you buy latency with runner minutes (Cost Awareness).
- Splitting checks into fast and slow tiers means the fast tier can pass on something the slow tier will reject, so an author may start the next change on a false green.
- Every quarantined test is coverage you are knowingly not enforcing. That is the right call against a flake, and it is still a real gap until it is fixed.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe relationship between discovery distance and correction cost is independent of stack. The absolute durations that count as "fast" differ hugely: a Go service and a large iOS app do not have the same achievable floor.
- TOOL-SPECIFICWhere the verdict is surfaced shapes whether it is read. GitHub renders checks inline on the PR; a Jenkins job that emails a link produces the same verdict with far more friction, and friction is latency.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — the test pyramid as an argument about feedback latency rather than about test counts.