Feature Flags and What They Cost
A flag decouples deploying code from releasing behaviour, which is genuinely valuable. It also multiplies the state space of the system and creates a cleanup obligation nobody is measured on.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
What does adding a flag do to the number of behaviours my system can exhibit, and who removes it?
Ship a redesigned checkout to 5% of traffic, then 25%, then everyone — with the ability to turn it off in seconds if conversion drops.
Wrap the new checkout in if (flags.newCheckout), ship it, and turn it up gradually. The flag costs one boolean and buys instant rollback; there is no reason to be careful about it.
The boolean is not one boolean. With fourteen flags the system has, in principle, sixteen thousand behaviours, and the tests exercise perhaps three of them (Boolean Flag Explosion).
- The boolean is not one boolean. With fourteen flags the system has, in principle, sixteen thousand behaviours, and the tests exercise perhaps three of them (Boolean Flag Explosion).
- Flags interact. The new checkout plus the old tax flag produces a combination nobody tried, and the bug report describes behaviour that does not reproduce because the reporter had a different flag set.
- The flag leaks inward. It starts at the entry point and, over three sprints, appears in the pricing module, the email templates and a database query, because each was easier to branch than to abstract (Invariant Leaks).
- Nobody removes it. It sits at 100% for a year, and now the old branch is untested code that nobody dares delete because they cannot prove it is dead (What "Legacy" Actually Means).
- As requirements arrive, every one must be considered against both branches, so the flag has converted a single codebase into two that must be maintained in parallel (Divergent Change).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The old and new checkout must both work against the same order data, because a customer may start in one and return in the other (Backward Compatibility as a Constraint).
- The flag must be evaluable server-side and client-side consistently, or a user sees one checkout and gets the other's pricing.
- The team already has fourteen flags in the codebase, four of which have been at 100% for over a year.
- Support needs to be able to tell, from a customer's order id, which variant that customer saw (Debuggability by Design).
- A given request must see exactly one variant, consistently, across every service that participates in it. A flag evaluated twice with different answers is the characteristic flag bug.
- Both branches must be correct — a flag is not a place to keep code that does not work yet.
- Every flag has an owner and a removal condition from the moment it is created. A flag without one is permanent by construction.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The flag owns exactly one decision — which variant this request gets — and it should own it at one point, as early in the request as possible.
- The evaluated value, not the flag, is what flows through the system: resolve once at the boundary and pass the decision down (Capability Passing).
- Each flag has a named owner and a removal condition, recorded with the flag itself rather than in a ticket.
- Someone owns the flag inventory as a number that must go down as well as up.
- Evaluate at the boundary — the request handler, the page load, the job start — and never inside domain logic. A domain rule that branches on a flag is a domain rule with two definitions (Where Invariants Live).
- The variant decision should be recorded on the artefact it affects: the order, the log line, the trace. That is what makes support's question answerable (Logging at Boundaries).
- The boundary between a release flag (temporary, removed) and an operational or permission toggle (permanent, part of the product) is worth drawing explicitly, because the second is not a migration device and should not be counted as debt.
The flag is not the cost; the combinations are
A single flag is obviously worth it, which is why the count grows. The cost is not in any one flag but in the number of configurations the system can be in, and that number is exponential in the count while every individual decision looks linear.
Price a routine change in the flagged area, and the mechanism becomes visible: the work is not the edit, it is the number of branch combinations someone has to think about before merging.
Apply a percentage discount code at checkout, affecting the displayed total, the charged amount and the confirmation email.
The change must work under new checkout and old, with the tax flag on and off, and with the express-shipping experiment running. Nine combinations are plausible; four are tested; the bug that ships reproduces only in one of the other five.
Pricing receives a CheckoutVariant value and never reads a flag. The combinations that matter are enumerable because there are two live flags, not fourteen, and the decision is recorded on the order so support can answer which variant a customer saw.
The flag that stopped being a migration device
The characteristic artefact is a flag at 100% for a year with both branches nominally supported. It is not obviously wrong — it works, the code is there, and turning it off is theoretically possible — which is exactly why it survives.
What makes it a problem is that it is *claimed* as a rollback while not functioning as one, and that every change in the area still pays for both branches.
looks like A boolean at 100% rollout for eleven months. The off branch still compiles, is still in the test suite, and was last executed in production the day before the rollout completed. Nobody knows who owns it, and its ticket is closed.
suggests The migration finished and its scaffolding was never removed. Every change in this area is still priced against two branches, and the rollback the flag supposedly provides has not been exercised since it stopped being needed.
fix Delete the branch that lost, not the flag check — that order matters, because deleting the check first leaves dead code with no marker. Then remove the flag definition, then the flag from the platform. Do it the week the rollout completes, when the argument about whether the old branch is dead has an obvious answer (Deprecation).
A flag has a lifecycle, and it must reach the end
Treating a release flag as a small state machine makes the missing transition obvious: almost every flag reaches FullyOn and almost none reaches Removed, and there is nothing in the system that notices.
The forbidden transitions are the ones that make a flag dangerous — going live with an untested branch, and deleting the definition while code still reads it.
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| Created | rollout begins | Ramping | both branches pass CI and the off branch is the current production behaviour | variant recorded on each affected order and log line |
| Ramping | metrics hold at each step | FullyOn | no regression in conversion or error rate across a full business cycle | old branch becomes idle but intact |
| Ramping | rollback | Created | nothing was written that the off branch cannot read (Backward Compatibility as a Constraint) | traffic returns; the flag has done its job |
| FullyOn | rollback | Ramping | off branch still tested and still correct — which decays with time | incident review |
| FullyOn | removal condition met | BranchDeleted | two weeks fully on with no rollback | losing branch and its tests deleted |
| BranchDeleted | cleanup | Removed | no code reads the flag | definition removed from code and platform |
- Created → FullyOn — Jumping to 100% skips the evidence the flag existed to gather. If you were always going to go straight to full traffic, the flag bought nothing except a state you now have to remove.
- FullyOn → Removed — Deleting the flag definition while both branches still exist in code leaves a branch selected by a default nobody chose. Delete the losing branch first, then the check, then the definition — in that order.
- Ramping → BranchDeleted — Deleting the old branch while the rollout is incomplete removes the rollback in the exact window where it is most likely to be needed.
The transition that does not happen is FullyOn -> BranchDeleted, and nothing in a normal system notices its absence. Make the removal condition machine-checkable and surface overdue flags somewhere people already look (The Debt Register).
How to build it
Most important first.
- Resolve the flag once, at the edge, into an explicit decision object that the rest of the code receives as data. This is the single design move that prevents flags from spreading.
- Give every flag a type: release, experiment, operational toggle, or permission. Only the first two are temporary, and conflating them is why "we have 200 flags" is an unanswerable statistic.
- Write the removal condition at creation time — "delete when at 100% for two weeks" — in the flag definition, and make an overdue flag visible somewhere people look (Revisit Triggers).
- Prefer flags that select between two *implementations of the same interface* over flags that branch inside a function. The first can be deleted by removing a line of wiring; the second requires reading every branch (Strategy).
- Test both branches in CI, and test the small number of combinations that can actually co-occur. Untested branches behind flags are the mechanism by which a rollback fails at the worst moment.
- Cap the number of live temporary flags per area, deliberately. Not because a number is principled, but because an unbounded count is how the state space becomes unreasonable (The Complexity Budget).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Adding the flag: hours, and it genuinely buys a rollback measured in seconds instead of a deploy. That value is real and this lesson is not arguing against it.
- Every subsequent change in the flagged area: more expensive, because it must be considered against both branches, and the increase persists for the flag's whole life.
- Removing the flag: half a day, and it is the cheapest it will ever be on the day the flag reaches 100%. Six months later it costs a day plus an argument about whether the old branch is really dead.
- The cost that compounds: the *n*-th flag is more expensive than the first, because the combinations to consider grow with the count rather than with the change (Change Amplification).
- Decoupling deploy from release is genuinely valuable and this lesson's caution should not be read as opposition: shipping dark and releasing gradually prevents a class of incident that no amount of testing does.
- Resolving at the edge into a decision object is more work than an
ifand is harder to retrofit than to start with. - A strict removal policy occasionally deletes something on-call wanted, which is the argument for typing flags rather than for keeping them all.
What can go wrong
- Combinatorial bugs: the defect exists only with flags A on and B off, so it does not reproduce for the person investigating and is closed as unreproducible.
- Inconsistent evaluation within one request — server says new, client says old — producing a user who sees one price and is charged another.
- Permanent flags: at 100% for a year, both branches nominally supported, one of them untested. The rollback that justified the flag no longer works.
- Flags used to hide unfinished work, so the off branch is not merely unused but broken, and the flag is now the only thing standing between production and a defect (Deliberate Debt).
- The mitigation fails on its own terms: a cleanup policy is adopted, flags are removed in bulk, and one of them was an operational toggle that on-call needed at 3am.
- Every flagged code path depends on the flag system's availability and latency. A flag service that is down should fail to a defined default, and that default is a design decision (Designing for Failure).
- Flags create a dependency between otherwise unrelated features, because their combinations must be considered together (Temporal Coupling).
- A client-evaluated flag depends on the client and server agreeing, which is a distributed-consistency problem hiding inside a boolean. Frontend Engineering owns the client-side evaluation mechanics; this lesson owns what the branch does to your code.
- "Flags let us ship unfinished work." They let you ship *finished* work that is not yet released. Code behind a flag that does not work is not hidden — it is one misconfiguration away from being live (Deliberate Debt).
- "A flag is a rollback." Only if the off branch is currently correct and tested. A year-old off branch is a hypothesis about a rollback.
- "Two flags mean four cases, which is fine." Four cases is fine; fourteen flags is sixteen thousand, and the growth is in the count of flags, not in the size of any change (Boolean Flag Explosion).
- "This is a delivery concern, not a design one." Delivery owns the rollout mechanics, and Backend and Frontend own evaluation and targeting. What a flag does to your state space, your invariants and your ability to reason locally is a design consequence, and it lands in the code you write (Local Reasoning).
- boolean-flag-explosion
- divergent-change
Testing it, and how it ages
- Test both branches — a flag with an untested off branch has no rollback, only the belief in one.
- Test the combinations that can genuinely co-occur, chosen deliberately. Exhaustive combination testing is impossible past about four flags and pretending otherwise wastes the budget (The Complexity Budget).
- Assert single evaluation per request: a test that the decision object is resolved once and passed, rather than re-read, catches the inconsistency class outright.
- Include the flag state in test failure output and in production logs, or every investigation starts by asking which variant this was (Stable Identifiers).
- Healthy flag use is a sawtooth: the count rises during releases and falls when they complete. A count that only rises is the signal, and it is visible long before anyone feels the pain.
- Flags that survive their purpose usually become one of two things — a permission or an operational toggle. Promoting them deliberately, with a new name and an owner, is much better than leaving a release flag in place and quietly relying on it (Deprecation).
- As the flagged code diverges, deleting the losing branch stops being a deletion and becomes a small migration of its own.
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a runtime branch multiplies reachable configurations, and that the branch must be removed by someone, holds for any flag mechanism from an environment variable to a targeting platform.
- SCALE-SPECIFICWith three flags and one deployable, combinations are enumerable and the cleanup burden is a chore. With two hundred flags across a dozen services evaluated client- and server-side, no one can state what the system does, and flag hygiene becomes a platform concern with its own tooling and owner. The technique is the same and the problem is not.
- CONTESTEDThe strongest opposing view: trunk-based development with continuous delivery genuinely depends on flags, and teams that restrict them end up with long-lived branches, big-bang merges and a far worse class of integration failure — a state space you can at least query beats a merge conflict you cannot. Practitioners who ship this way argue the cleanup burden is a tooling problem, solved by expiry dates and automated pull requests, not a reason for fewer flags. That is a strong case, and it is compatible with everything here except the habit of leaving flags at 100% for a year (Trunk-Based Development in DevOps).
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — choosing which flag combinations to test, and using production experiments as evidence, is a statistics and coverage question this lesson only frames as a design cost.