CI/CD Anti-Patterns
Seven pipeline habits that quietly convert a feedback system into a bottleneck you cannot trust, and the pressure that produces each.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What turns a delivery pipeline from a safety mechanism into a ritual people route around?
A pipeline exists to give fast, trustworthy feedback and to produce an artifact you can trace and reverse. Each of these habits removes one of those properties while leaving the pipeline apparently intact — it still runs, still goes green, and no longer tells you anything.
Get the pipeline working, then leave it alone. It runs on every commit, it goes green, and time spent on it is time not spent on features.
Pipeline duration creeps upward one job at a time. Nobody adds thirty minutes; six people add five (CI Is a Feedback System).
- Pipeline duration creeps upward one job at a time. Nobody adds thirty minutes; six people add five (CI Is a Feedback System).
- Once feedback is slower than a context switch, people stop waiting for it, and the pipeline stops being a gate in practice while still being one on paper.
- A suite with known flaky tests trains everyone to re-run rather than read, so a genuine regression gets re-run to green (Flaky Tests).
- A pipeline that rebuilds per environment tests one thing and ships another, so the green tick is evidence about an artifact that no longer exists.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A pipeline provides two things: fast feedback and a trustworthy artifact. Every anti-pattern here destroys one of them.
- Feedback value decays with delay. At two minutes it is part of writing the change; at sixty it arrives after you have moved on, and the cost of acting on it includes reloading everything you knew.
- Trust is binary and asymmetric. A suite that is right 95% of the time is not 95% as useful — a single accepted false failure teaches the whole team to discount every failure, and a green build now means "probably".
- Artifact trustworthiness comes from identity and provenance: this exact digest, built from that commit, by that pipeline, and promoted rather than rebuilt. Break any link and the deployment record is no longer evidence (Build Provenance).
- The final link is verification. A pipeline that deploys and reports success on "the deploy command exited zero" has automated the action but not the outcome (A Successful Deploy Is Not Evidence of a Healthy System).
The seven
Each of these arrives for a reason, and the reason is usually a good one at the moment it arrives. That is what makes them hard to see: the pipeline is not neglected, it is locally optimised at every step.
| Anti-pattern | Why it is tempting | What it destroys | Instead |
|---|---|---|---|
| One 60-minute pipeline | Every job was added for a good reason, and each addition was small | Feedback value. People batch changes, context-switch away, and stop treating it as a gate (CI Is a Feedback System) | Fast checks first, expensive ones in parallel, slow suites off the blocking path (Parallelising CI) |
| Flaky tests tolerated | Fixing them is unglamorous and re-running is one click away | Trust in every result. Red stops meaning red, and a real regression gets re-run to green | Quarantine on detection, with an owner and a deadline (Flaky Tests) |
| No artifact immutability | Mutable tags are convenient and nobody wants to manage digests by hand | The identity of what is running. Two deploys of "the same" version can differ (Tags Versus Digests) | Content-addressed digests, immutable registries, tags as labels only |
| No build provenance | It works without it, and attestation is extra machinery for no visible benefit | The link from running artifact back to source. During an incident you cannot prove what is in it (Build Provenance) | Record commit, inputs, builder and digest as a signed attestation (Signing and Verifying Artifacts) |
| No environment promotion | Rebuilding per environment is simpler to set up and looks equivalent | The meaning of testing. What passed staging is not what production runs (Promotion Between Environments) | Promote the identical artifact; vary only configuration (Build Once, Deploy Many) |
| No deployment verification | The deploy command succeeded, so the deploy succeeded | The difference between action and outcome. A healthy exit code says nothing about error rate (A Successful Deploy Is Not Evidence of a Healthy System) | Verify against health and error signals before declaring success; roll back automatically on clear regression |
| No rollback path | It has never been needed, and forward-only sounds disciplined | Your options during an incident. Reversal becomes an improvisation at the worst moment (Rollback: Only Useful If It Is Actually Safe) | A rehearsed, one-command reversal with a known duration |
Where the time actually goes
Before restructuring a slow pipeline, measure it. The instinct is to parallelise everything, which adds compute cost and, on jobs that share mutable state, converts slowness into flakiness — swapping a known problem for a worse one.
The ordering principle is cheap and decisive: run the checks that fail most often and cost least first, so the common failure is reported in a minute rather than in an hour.
Which intervention fits the shape of the slowness you measured?
when Most failures come from lint, type checks or unit tests that currently run after a long build
cost Nearly free, but only helps the failing case — a green run still takes 60 minutes
when The dependency graph is genuinely wide and jobs do not share mutable state (The CI Dependency Graph)
cost More concurrent compute, and any shared state turns into intermittent failures
when A large share of time is re-fetching or rebuilding unchanged inputs
cost Cache correctness becomes a real concern; a stale hit can hide a dependency change (Caching in CI)
when End-to-end or performance suites dominate and rarely catch what earlier stages miss
cost You are choosing not to gate on that coverage; it must still run somewhere with an owner for failures
when A monorepo runs everything for every change (The CI Dependency Graph)
cost Change-detection logic is now load-bearing, and getting it wrong skips checks silently
when The suite is genuinely necessary and irreducible, e.g. hardware-in-the-loop or compliance runs
cost Requires strong verification and fast rollback downstream, since feedback will arrive late regardless (Continuous Delivery)
A pipeline whose green tick means something
The properties are cumulative: each stage is only meaningful if the one before it preserved identity. Promotion without immutability promotes nothing in particular, and verification without promotion verifies a different artifact than the one that was tested.
- 1Commit
Establishes the source of truth for this change, with review and required checks (Required Checks).
fails by Direct pushes and bypassed checks make the commit history an unreliable record.
evidence A protected branch where every merge carries a review and a passing check set.
- 2Fast checks
Lint, types, unit tests — the cheap, high-yield failures, within a couple of minutes.
fails by Ordered after the build, so the cheapest failure costs the most time.
evidence Median time-to-first-failure measured in minutes, not tens of minutes.
- 3Build
Produces one artifact from pinned inputs and records its digest.
fails by Unpinned inputs make the build non-reproducible, so the same commit yields different bits (Reproducible Builds).
evidence Two builds of the same commit yield the same digest, or the differences are known and explained.
- 4Provenance
Attaches a signed record of commit, inputs and builder to the digest.
fails by Provenance generated outside the builder, so it attests to a claim rather than an observation (The Builder Is Inside the Trust Boundary).
evidence An attestation verifiable at deploy time, not merely stored alongside the artifact.
- 5Slow checks
Integration and end-to-end suites, run against that exact artifact.
fails by Run against a rebuild, so they test something other than what ships.
evidence Test logs naming the digest under test, matching the digest that will be promoted.
- 6Promote
Moves the identical artifact toward production; only configuration varies (Promotion).
fails by A rebuild at promotion silently substitutes a different artifact.
evidence The digest in production equals the digest that passed the slow checks.
- 7Deploy and verify
Rolls out, then compares health, error rate and latency against a baseline before declaring success.
fails by Success declared on the deploy command's exit code (Canary Analysis: Compared Against What?).
evidence A recorded verification result per deployment, with the thresholds it was judged against.
- 8Reverse
Returns to the previous digest on a single command, in a known time.
fails by Never exercised, so its duration and correctness are unknown (Rollback: Only Useful If It Is Actually Safe).
evidence A recent real rollback with a measured duration.
Read it backwards to audit an existing pipeline: if you cannot show the digest in production is the one that passed the slow checks, everything upstream of that break is decoration.
How to do it properly
Most important first.
- Set a target for feedback latency and defend it. Fast checks first, expensive checks in parallel, slow suites off the critical path and into post-merge or scheduled runs (Designing the Pipeline).
- Treat a flaky test as a broken test: quarantine it out of the gating path immediately, with an owner and a deadline, so the suite stays trustworthy while it is fixed.
- Build once, address by digest, promote the same artifact through environments (Promotion).
- Record provenance — commit, inputs, builder, and the resulting digest — as a signed record attached to the artifact (Build Provenance).
- Verify deployments against health and error signals before calling them successful, and make rollback a first-class pipeline action rather than a manual runbook step.
- Cache aggressively but correctly, and measure cache hit rate; a cache that silently stopped hitting is a common source of slow-pipeline drift (Caching in CI).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A degraded pipeline is upstream of every change, so its faults reach production through all of them. What contains it is verification and rollback at the far end.
What can go wrong
- Parallelising a pipeline whose jobs share mutable state, converting a slow pipeline into a fast flaky one.
- Quarantine used as a permanent home, so coverage decays silently and the quarantine list becomes the real test suite.
- Caching that hides a real dependency change, so a build succeeds against stale inputs and reproducibility is lost (Reproducible Builds).
- Promotion machinery so heavy that people ship urgent fixes around it, which is the pipeline being routed around in its most dangerous form.
- Automated verification with thresholds set so loosely that every deploy passes, which is worse than no verification because it is believed.
- "A green pipeline means the change is safe." It means the checks you wrote passed on the artifact you built. Whether that artifact behaves in production is what verification is for.
- "Flaky tests are a test-quality problem." They are a delivery problem: they set the team's response to a red build, which is the thing you actually rely on.
- "Slow pipelines are only a productivity issue." They change behaviour — batching, skipping, out-of-band hotfixes — and every one of those is a reliability issue (Change Size: Why Small Changes Are Safer, and When They Are Not).
- "Continuous deployment requires a perfect pipeline." It requires a trustworthy one plus a fast reversal. Perfection is neither achievable nor the mechanism (Continuous Deployment).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Median and p95 pipeline duration are graphed and have a target, rather than being a thing people complain about.
- The re-run rate on the main branch is low, and a red build gets read rather than re-run.
- The digest running in production is byte-identical to the one that passed tests, and you can show the chain.
- A rollback has been executed through the pipeline recently and took a known number of minutes.
- Pipeline changes are themselves deployable and should be reversible: keep the pipeline definition in version control alongside the code it builds.
- When a new gate turns out to block more than it catches, remove it deliberately and record the reasoning, rather than granting standing bypasses that quietly become the norm.
- Automate flake detection — repeated runs on unchanged commits, quarantine on a threshold, and an owner assigned automatically.
- Automate provenance and digest propagation so nothing depends on a person copying a version string between environments.
- Automate post-deploy verification and automatic rollback on clear regressions; keep the decision to override the gate human (Guardrails, Not Gates).
- Faster feedback usually means more parallelism and more compute cost, and a portion of the suite moved off the blocking path — which is real coverage you are choosing not to gate on.
- Promotion and provenance add steps between merge and production, which is exactly the latency they are supposed to be protecting you from. Keep the machinery thin.
- Automatic rollback on regression will sometimes reverse a good deploy on a noisy signal, and each false rollback costs the team confidence in the gate.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe seven failures apply to any delivery pipeline regardless of CI system. What differs is which are easy to fix: hosted systems make parallelism and caching a configuration change, while self-hosted runners make them a capacity problem.
- TOOL-SPECIFICArtifact immutability and provenance depend on registry and CI features — content-addressed digests, attestation support, signed provenance. Where the tooling lacks them, the property must be enforced by convention and is correspondingly weaker (The Builder Is Inside the Trust Boundary).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — which suites belong on the blocking path at all, and how test value is measured rather than assumed.