Infrastructure Fundamentals

No Cargo-Cult Infrastructure

Complexity must be justified by the workload and by the organization that has to run it. Adopting the architecture of a company a thousand times your size gives you their operational burden and none of their problems.

▶ Run the lab

The question this answers

Infrastructure question

How do I tell infrastructure that solves a real problem from infrastructure that was adopted because it is what serious companies use?

Application requirement

A five-engineer team runs one API at 40 requests per second with a five-gigabyte database. A design review proposes a managed Kubernetes cluster, a service mesh, a streaming platform and a multi-region database.

What it provides

A test that can be applied in a design review: for each component, the concrete problem that forced it, the operational burden it adds, and who on this team will carry that burden at 03:00.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

The same workload, twice

Cargo-cult infrastructure is not stupidity. It is a rational response to bad incentives: reference architectures are written for the largest plausible reader, conference talks describe problems at a scale almost nobody has, and "we used the boring option" is harder to defend in a review than "we used the industry standard". The result is teams operating systems designed for problems they do not have.

The comparison below is one real workload — a B2B API, 40 requests per second at peak, a five-gigabyte database, five engineers, no on-call rotation worth the name. The left column is a design that would pass most reviews. The right column serves the same traffic with the same availability, and can be understood in an afternoon by someone who has just joined.

The decisive difference is not the number of boxes. It is that every line on the right names the problem that created it, and every line on the left would need one invented. Ask "what breaks if we remove this?" of each left-hand component and most answers are "nothing, today" — which means each of them is a monthly bill and a share of the team's attention purchased against a hypothetical.

Over-built: 40 req/s, 5 engineers, 5 GB of data
platform:
  kubernetes: managed, 3 node groups, 2 environments
  service_mesh: sidecar per pod, mTLS, traffic policies
  ingress: controller + cert-manager + external-dns
  gitops: reconciliation operator + its own repo
services:
  count: 6            # split from one app, by one team
  communication: gRPC + mesh retries + circuit breakers
data:
  postgres: multi-region, async replica in a second continent
  kafka: 3 brokers   # one topic: "user.registered"
  redis: 3-node cluster  # no measured cache-miss problem
observability:
  stack: self-hosted metrics + logs + traces + dashboards
# on-call: two people, neither has operated a mesh before
Right-sized: the same traffic, the same availability target
compute:
  managed_container_service: 2 tasks, 2 zones
  # forced by: one restart must not be a visible outage
entry:
  load_balancer: TLS termination, automated cert renewal
  # forced by: a stable address that survives task replacement
data:
  postgres: managed, single region, standby in zone B
  # forced by: losing customer data is unacceptable;
  #            a zone event must not lose the write path
  backups: automated + restore drill every quarter
  object_storage: user uploads
  # forced by: files vanished when a task was replaced
async:
  jobs_table + worker task    # replaces Kafka for one topic
  # forced by: the welcome email must survive a crash
observability:
  managed: metrics, logs, 4 alerts that page a human
  # forced by: nobody is watching at 03:00
# on-call: two people, both can explain every box above

Both designs meet the same availability target for the same traffic. The right-hand one can be operated by the team that has to operate it, which is a technical property and not a soft one — infrastructure nobody has capacity to run is infrastructure that will be run badly, patched late and misconfigured under pressure. The left-hand design also fails *differently*: a mesh misconfiguration or a control-plane upgrade is an outage class the right-hand design simply does not have.

The five questions, and the honest scoring

The test is five questions per component, asked in a design review, with answers written down. It is deliberately awkward, because the awkwardness is the mechanism: a component that cannot survive being asked "who on this team has operated this before?" is a component whose adoption is a bet on learning during an incident.

What makes this work is scoring the *organization* alongside the technology. Kubernetes at a company with a platform team of eight is a reasonable choice that pays for itself. The identical Kubernetes at a company with five product engineers and no platform team is a different decision with a different expected outcome, using the same software. The technology is not the variable; the operator is.

One clarification the anti-complexity argument needs, or it becomes its own cargo cult: simple is not the same as manual. Infrastructure as code, automated backups, a CI pipeline and real alerting all add components and are almost always worth it, because they reduce the number of things a human must do correctly under pressure. The test is not "how many boxes" — it is "how much does this reduce the failure modes we actually have, against how much attention does it demand".

  • Score the organization, not just the technology: the same component is a good decision for one team and a bad one for another.
  • Simple is not manual. IaC, automated backups, CI and alerting add components and reduce the number of things humans must get right — adopt them early.
  • Complexity is asymmetric. Kubernetes arrives in a sprint and leaves over two quarters, because removal requires proving nothing depends on it.
  • A component adopted "for later" is paid for now, in money and attention, at the moment when both are scarcest.
  • The strongest signal of cargo culting is that nobody can name the forcing event — see From a Simple App to a Scaled System in the architecture domain for the same test applied one level up.
QuestionA good answer looks likeA bad answer costs you
What concrete problem forced this?"Deploys dropped in-flight requests; we measured 400 failed requests per release.""It is best practice" — a monthly bill and an attention budget spent on a hypothetical
What is the simplest thing that solves it?"A second instance behind the existing load balancer."A distributed system adopted where a table and a cron job would have done
What does this add to the operational surface?"A control plane to upgrade quarterly, and a new class of scheduling failure."A 03:00 page about a component nobody on the rotation understands
Who here has run this in production?"Nobody — so we are choosing the managed version and accepting the constraint."Learning a new failure domain during your first incident in it
What triggers removing it again?"If the queue depth stays under 10 for two quarters, we delete the worker tier."Permanence by default; nothing is ever removed because nobody can prove it is unused
Five questions per component. The right-hand column is what a bad answer actually costs.

What the over-built design actually costs

The financial cost is the visible half and usually not the largest. A control plane, three brokers and a three-node cache tier bill by the hour regardless of traffic, which for a 40 req/s workload means the fixed baseline dwarfs everything the traffic drives. That is straightforwardly measurable and straightforwardly embarrassing.

The larger cost is attention. Every component is an upgrade cadence, a set of failure modes, a body of documentation and a share of onboarding. Five engineers operating twelve components have roughly two and a half components each, and the practical result is that most components are operated by nobody: not patched, not tuned, not monitored beyond the default dashboard, and understood by exactly one person who is currently on holiday.

And there is a third cost that only appears during an incident: novel failure modes. The over-built design can fail in ways the simple one cannot — a mesh certificate rotation failing, a control-plane upgrade evicting pods faster than they can be rescheduled, a broker partition leader election stalling. Those outages are not caused by traffic. They are caused by having the component at all, which means the complexity did not just fail to help availability; it actively spent some.

The over-built design at 40 req/s. Note how little of the bill the traffic drives. Relative weights.ILLUSTRATIVE
Orchestrator control plane + nodes fixed
driven by cluster-hours + node-hours, independent of traffic · Identical bill at 40 req/s and 40,000.
Service mesh overhead · surprisefixed
driven by a sidecar per pod: CPU, memory, and a second network hop · Consumes resources on every pod and adds latency to every internal call.
Broker cluster · surprisefixed
driven by broker-hours + storage for one low-volume topic
Cache cluster · surprisefixed
driven by node-hours with no measured cache-miss problem
Cross-region replication · surpriseusage
driven by GB replicated continuously to a second continent · The highest per-GB rate on most price lists, running 24/7 for a hypothetical.
Traffic-driven compute usage
driven by the actual 40 req/s · The only line the workload moves. It is the smallest one on the list.
Engineering attention · surprisefixed
driven by components ÷ engineers, weighted by novelty · Twelve components across five engineers means most are owned by nobody.

Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.

Key points

  • Every component must name the concrete problem that forced it; "best practice" and "we will need it later" are not forcing events.
  • Score the organization as well as the technology — the same component is a sound choice for a team with a platform group and a liability for a team without one.
  • Simple is not manual: infrastructure as code, backups, CI and alerting add components while reducing what humans must get right, and are worth adopting early.
  • The dominant cost of over-building is attention, not money, and its most damaging effect is failure modes the simple design does not have.
  • Complexity is asymmetric: added in a sprint, removed over quarters, because removal requires proving a negative.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • For each proposed component, record the forcing event with a measurement attached, the simplest alternative considered, and the removal trigger.
  • Weight the operational burden by the team's existing experience — a component nobody has run carries a multiplier, not a flat cost.
  • Sum the burden across components and compare it to the number of people who will actually carry it, including holidays and turnover.
  • Re-run the review annually: components stay long after their forcing event has gone, and nobody notices because nothing breaks when they are ignored.
What you still own
  • The removal triggers, which only work if someone actually checks them — an unchecked trigger is a comment, not a control.
  • An honest inventory of who understands each component, refreshed when people leave, because single-person knowledge is the real availability risk.
  • The upgrade cadence of every adopted component, which is the recurring cost people forget when adopting and remember when a version reaches end of life.
How it fails
  • An outage caused by a component that exists only to prevent outages — a mesh certificate rotation, a control-plane upgrade, a broker leader election.
  • A component silently unowned: not patched, not monitored, understood by one engineer who has left.
  • A team that cannot ship, because a two-line change requires touching six services and four pipelines built for an organization with six teams.
  • A cost review that finds most of the bill is fixed baseline for components the traffic never touches, and no clear path to removing any of them.
How it scales
  • Over-built systems do not scale better; they scale the same and cost more, because the constraint at small scale was never the thing the complexity addressed.
  • What genuinely does not scale is attention: components grow linearly and engineers do not, so past roughly two components per engineer, quality per component starts falling.
  • The right time to adopt complexity is when the simple thing has visibly failed at your current scale — that failure is both the justification and the specification.
Security
  • Every component is attack surface: a control plane, a mesh, a broker and a registry each have their own authentication, their own CVEs and their own upgrade obligation.
  • Unowned components are unpatched components, and an unpatched control plane is a far more attractive target than an unpatched application.
  • More components means more identities and more policies, which makes least privilege harder to achieve and easier to quietly abandon — see Least Privilege in Infrastructure.
Cost shape
  • Over-building shows up as a large fixed baseline that does not move with traffic — the diagnostic signature is a bill that barely changes when usage halves.
  • The attention cost is invisible on invoices and is usually the largest single line if you price engineering time honestly.
  • Removal has its own cost, which is why the correct moment to say no is at adoption and not later.
What to watch
  • Utilization per component: near-zero utilization on a component with a non-zero bill is the clearest possible evidence of an unforced adoption — see Idle Capacity: Headroom or Waste?.
  • The ratio of fixed to usage-shaped spend, tracked over time; a rising fixed share means complexity is accumulating faster than traffic.
  • Incident causes tagged by component. If a meaningful share of your outages come from components adopted for reliability, the complexity is net negative.
  • The signal that lies: a fully green dashboard on an unused component. Health confirms it is running, not that anything needs it.
Simpler alternatives
  • A managed application platform or a single managed container service instead of an orchestrator. It provides deployment, health checks, rolling updates and scaling with no control plane of yours — see Kubernetes Is Not Always Needed.
  • A database table plus a worker loop instead of a message broker. It is transactional with your data, it is one fewer system, and it comfortably handles volumes far above what most teams adopting brokers actually have — see Message Queues in the architecture domain for when that stops being true.
  • A modular monolith instead of six services, when there is one team. Deployment independence is a benefit for multiple teams and pure overhead for one — see Modular Monolith.
  • Vertical scaling instead of a distributed data layer. A single large instance handles a surprising amount of load, and one machine has no consistency problem — see One Big VM or Several Small Ones.
  • Doing nothing yet, and writing down the measurement that would change the answer.
What adopting this costs
  • Right-sizing buys operability, comprehension and a smaller failure surface, and charges a later migration if the workload genuinely grows into the complexity.
  • That migration is real and should be acknowledged — but it is paid once, with knowledge you did not have at the start, rather than continuously against a guess.
  • Refusing complexity has a social cost: the smaller design looks less impressive in a review, and someone has to be willing to defend it.

Operational complexity, scored

Operational complexity, scored
Tick what your infrastructure actually contains. The number is deliberately crude — what matters is which drivers it names and whether the workload ever asked for them.
complexity
0
on-call surface
-1
upgrade cadence
-1
debugging difficulty
-1
expertise breadth required1 · distinct specialisms
Verdict
One engineer can hold this
Small enough that one person can describe every component from memory and a new hire is productive in a week.
Top drivers
+1 Self-managed VMs — You own the OS, the patch cadence and the drift.
Nothing ticked, or only the reducers: this is the baseline every system should be compared against. The question this widget exists to force is not "how do we manage our complexity" but "which component here was demanded by the workload, and which arrived because a conference talk made it sound inevitable?"
SIMULATEDthe weights are invented for teaching. They are not an industry metric and no real system should be judged by this number alone.

What people believe, and what is true

Claim

Building it simply now means an expensive rewrite later.

Reality

Sometimes. That rewrite is paid once, with real knowledge of the workload, and is usually cheaper than years of operating and paying for complexity aimed at a guess.

Claim

This is an argument against automation and tooling.

Reality

The opposite. IaC, CI, automated backups and alerting reduce what humans must get right under pressure. The target is unforced *distributed systems*, not unforced *automation*.

Claim

Big companies use this, so it must be the safe default.

Reality

They have the problem, the platform team and the operational maturity. You inherit the burden without any of the three, and the burden is the part that pages you.

Apply it