The question this answers
How do I tell infrastructure that solves a real problem from infrastructure that was adopted because it is what serious companies use?
A five-engineer team runs one API at 40 requests per second with a five-gigabyte database. A design review proposes a managed Kubernetes cluster, a service mesh, a streaming platform and a multi-region database.
A test that can be applied in a design review: for each component, the concrete problem that forced it, the operational burden it adds, and who on this team will carry that burden at 03:00.
The same workload, twice
Cargo-cult infrastructure is not stupidity. It is a rational response to bad incentives: reference architectures are written for the largest plausible reader, conference talks describe problems at a scale almost nobody has, and "we used the boring option" is harder to defend in a review than "we used the industry standard". The result is teams operating systems designed for problems they do not have.
The comparison below is one real workload — a B2B API, 40 requests per second at peak, a five-gigabyte database, five engineers, no on-call rotation worth the name. The left column is a design that would pass most reviews. The right column serves the same traffic with the same availability, and can be understood in an afternoon by someone who has just joined.
The decisive difference is not the number of boxes. It is that every line on the right names the problem that created it, and every line on the left would need one invented. Ask "what breaks if we remove this?" of each left-hand component and most answers are "nothing, today" — which means each of them is a monthly bill and a share of the team's attention purchased against a hypothetical.
platform: kubernetes: managed, 3 node groups, 2 environments service_mesh: sidecar per pod, mTLS, traffic policies ingress: controller + cert-manager + external-dns gitops: reconciliation operator + its own repo services: count: 6 # split from one app, by one team communication: gRPC + mesh retries + circuit breakers data: postgres: multi-region, async replica in a second continent kafka: 3 brokers # one topic: "user.registered" redis: 3-node cluster # no measured cache-miss problem observability: stack: self-hosted metrics + logs + traces + dashboards # on-call: two people, neither has operated a mesh before
compute: managed_container_service: 2 tasks, 2 zones # forced by: one restart must not be a visible outage entry: load_balancer: TLS termination, automated cert renewal # forced by: a stable address that survives task replacement data: postgres: managed, single region, standby in zone B # forced by: losing customer data is unacceptable; # a zone event must not lose the write path backups: automated + restore drill every quarter object_storage: user uploads # forced by: files vanished when a task was replaced async: jobs_table + worker task # replaces Kafka for one topic # forced by: the welcome email must survive a crash observability: managed: metrics, logs, 4 alerts that page a human # forced by: nobody is watching at 03:00 # on-call: two people, both can explain every box above
Both designs meet the same availability target for the same traffic. The right-hand one can be operated by the team that has to operate it, which is a technical property and not a soft one — infrastructure nobody has capacity to run is infrastructure that will be run badly, patched late and misconfigured under pressure. The left-hand design also fails *differently*: a mesh misconfiguration or a control-plane upgrade is an outage class the right-hand design simply does not have.
The five questions, and the honest scoring
The test is five questions per component, asked in a design review, with answers written down. It is deliberately awkward, because the awkwardness is the mechanism: a component that cannot survive being asked "who on this team has operated this before?" is a component whose adoption is a bet on learning during an incident.
What makes this work is scoring the *organization* alongside the technology. Kubernetes at a company with a platform team of eight is a reasonable choice that pays for itself. The identical Kubernetes at a company with five product engineers and no platform team is a different decision with a different expected outcome, using the same software. The technology is not the variable; the operator is.
One clarification the anti-complexity argument needs, or it becomes its own cargo cult: simple is not the same as manual. Infrastructure as code, automated backups, a CI pipeline and real alerting all add components and are almost always worth it, because they reduce the number of things a human must do correctly under pressure. The test is not "how many boxes" — it is "how much does this reduce the failure modes we actually have, against how much attention does it demand".
- Score the organization, not just the technology: the same component is a good decision for one team and a bad one for another.
- Simple is not manual. IaC, automated backups, CI and alerting add components and reduce the number of things humans must get right — adopt them early.
- Complexity is asymmetric. Kubernetes arrives in a sprint and leaves over two quarters, because removal requires proving nothing depends on it.
- A component adopted "for later" is paid for now, in money and attention, at the moment when both are scarcest.
- The strongest signal of cargo culting is that nobody can name the forcing event — see From a Simple App to a Scaled System in the architecture domain for the same test applied one level up.
| Question | A good answer looks like | A bad answer costs you |
|---|---|---|
| What concrete problem forced this? | "Deploys dropped in-flight requests; we measured 400 failed requests per release." | "It is best practice" — a monthly bill and an attention budget spent on a hypothetical |
| What is the simplest thing that solves it? | "A second instance behind the existing load balancer." | A distributed system adopted where a table and a cron job would have done |
| What does this add to the operational surface? | "A control plane to upgrade quarterly, and a new class of scheduling failure." | A 03:00 page about a component nobody on the rotation understands |
| Who here has run this in production? | "Nobody — so we are choosing the managed version and accepting the constraint." | Learning a new failure domain during your first incident in it |
| What triggers removing it again? | "If the queue depth stays under 10 for two quarters, we delete the worker tier." | Permanence by default; nothing is ever removed because nobody can prove it is unused |
What the over-built design actually costs
The financial cost is the visible half and usually not the largest. A control plane, three brokers and a three-node cache tier bill by the hour regardless of traffic, which for a 40 req/s workload means the fixed baseline dwarfs everything the traffic drives. That is straightforwardly measurable and straightforwardly embarrassing.
The larger cost is attention. Every component is an upgrade cadence, a set of failure modes, a body of documentation and a share of onboarding. Five engineers operating twelve components have roughly two and a half components each, and the practical result is that most components are operated by nobody: not patched, not tuned, not monitored beyond the default dashboard, and understood by exactly one person who is currently on holiday.
And there is a third cost that only appears during an incident: novel failure modes. The over-built design can fail in ways the simple one cannot — a mesh certificate rotation failing, a control-plane upgrade evicting pods faster than they can be rescheduled, a broker partition leader election stalling. Those outages are not caused by traffic. They are caused by having the component at all, which means the complexity did not just fail to help availability; it actively spent some.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Every component must name the concrete problem that forced it; "best practice" and "we will need it later" are not forcing events.
- Score the organization as well as the technology — the same component is a sound choice for a team with a platform group and a liability for a team without one.
- Simple is not manual: infrastructure as code, backups, CI and alerting add components while reducing what humans must get right, and are worth adopting early.
- The dominant cost of over-building is attention, not money, and its most damaging effect is failure modes the simple design does not have.
- Complexity is asymmetric: added in a sprint, removed over quarters, because removal requires proving a negative.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • For each proposed component, record the forcing event with a measurement attached, the simplest alternative considered, and the removal trigger.
- • Weight the operational burden by the team's existing experience — a component nobody has run carries a multiplier, not a flat cost.
- • Sum the burden across components and compare it to the number of people who will actually carry it, including holidays and turnover.
- • Re-run the review annually: components stay long after their forcing event has gone, and nobody notices because nothing breaks when they are ignored.
- • The removal triggers, which only work if someone actually checks them — an unchecked trigger is a comment, not a control.
- • An honest inventory of who understands each component, refreshed when people leave, because single-person knowledge is the real availability risk.
- • The upgrade cadence of every adopted component, which is the recurring cost people forget when adopting and remember when a version reaches end of life.
- • An outage caused by a component that exists only to prevent outages — a mesh certificate rotation, a control-plane upgrade, a broker leader election.
- • A component silently unowned: not patched, not monitored, understood by one engineer who has left.
- • A team that cannot ship, because a two-line change requires touching six services and four pipelines built for an organization with six teams.
- • A cost review that finds most of the bill is fixed baseline for components the traffic never touches, and no clear path to removing any of them.
- • Over-built systems do not scale better; they scale the same and cost more, because the constraint at small scale was never the thing the complexity addressed.
- • What genuinely does not scale is attention: components grow linearly and engineers do not, so past roughly two components per engineer, quality per component starts falling.
- • The right time to adopt complexity is when the simple thing has visibly failed at your current scale — that failure is both the justification and the specification.
- • Every component is attack surface: a control plane, a mesh, a broker and a registry each have their own authentication, their own CVEs and their own upgrade obligation.
- • Unowned components are unpatched components, and an unpatched control plane is a far more attractive target than an unpatched application.
- • More components means more identities and more policies, which makes least privilege harder to achieve and easier to quietly abandon — see Least Privilege in Infrastructure.
- • Over-building shows up as a large fixed baseline that does not move with traffic — the diagnostic signature is a bill that barely changes when usage halves.
- • The attention cost is invisible on invoices and is usually the largest single line if you price engineering time honestly.
- • Removal has its own cost, which is why the correct moment to say no is at adoption and not later.
- • Utilization per component: near-zero utilization on a component with a non-zero bill is the clearest possible evidence of an unforced adoption — see Idle Capacity: Headroom or Waste?.
- • The ratio of fixed to usage-shaped spend, tracked over time; a rising fixed share means complexity is accumulating faster than traffic.
- • Incident causes tagged by component. If a meaningful share of your outages come from components adopted for reliability, the complexity is net negative.
- • The signal that lies: a fully green dashboard on an unused component. Health confirms it is running, not that anything needs it.
- • A managed application platform or a single managed container service instead of an orchestrator. It provides deployment, health checks, rolling updates and scaling with no control plane of yours — see Kubernetes Is Not Always Needed.
- • A database table plus a worker loop instead of a message broker. It is transactional with your data, it is one fewer system, and it comfortably handles volumes far above what most teams adopting brokers actually have — see Message Queues in the architecture domain for when that stops being true.
- • A modular monolith instead of six services, when there is one team. Deployment independence is a benefit for multiple teams and pure overhead for one — see Modular Monolith.
- • Vertical scaling instead of a distributed data layer. A single large instance handles a surprising amount of load, and one machine has no consistency problem — see One Big VM or Several Small Ones.
- • Doing nothing yet, and writing down the measurement that would change the answer.
- • Right-sizing buys operability, comprehension and a smaller failure surface, and charges a later migration if the workload genuinely grows into the complexity.
- • That migration is real and should be acknowledged — but it is paid once, with knowledge you did not have at the start, rather than continuously against a guess.
- • Refusing complexity has a social cost: the smaller design looks less impressive in a review, and someone has to be willing to defend it.
Operational complexity, scored
What people believe, and what is true
Building it simply now means an expensive rewrite later.
Sometimes. That rewrite is paid once, with real knowledge of the workload, and is usually cheaper than years of operating and paying for complexity aimed at a guess.
This is an argument against automation and tooling.
The opposite. IaC, CI, automated backups and alerting reduce what humans must get right under pressure. The target is unforced *distributed systems*, not unforced *automation*.
Big companies use this, so it must be the safe default.
They have the problem, the platform team and the operational maturity. You inherit the burden without any of the three, and the burden is the part that pages you.