The question this answers
How much operational complexity has this architecture actually committed the team to, and which parts of it does the workload require?
An internal dashboard used by about five hundred employees, read-heavy, tolerant of a few minutes of downtime, maintained by three engineers who also have other responsibilities. Its current design has multi-region Kubernetes, a service mesh, a Kafka cluster, a Redis cluster and a document database alongside the relational one.
A shared, comparable number for what a design costs to operate — not a measure of quality, but a way to make an implicit ongoing commitment explicit at the moment it is being made.
A crude score, on purpose
Complexity arguments go nowhere because both sides are describing feelings. One engineer says the design is over-engineered; another says it is production-grade; nothing is measurable and the louder person wins. A rough score fixes the conversation by giving it units. Each significant infrastructure choice contributes points; the points are added; the total is compared against the team size and the workload's actual requirements. The number is not the point — the conversation it forces is.
The weights below are a teaching device produced for this app, not an industry constant. Kubernetes is not "worth exactly five" of anything; the weight encodes a judgement that adopting an orchestrator adds substantially more ongoing operational surface than adding a virtual machine, and that a service mesh adds more still because it inserts itself into every request path. Your own weights may differ, and a team that has run Kubernetes for six years should score it lower than a team adopting it next month — because the real variable is not the technology, it is the gap between what the system demands and what the team already knows.
Use it comparatively. Score the design you have, score the simplest design that meets the requirement, and look at the difference. If the difference is large and the requirement did not change, you have found complexity that arrived for reasons other than the workload — which is the entire argument of No Cargo-Cult Infrastructure made numerical.
| Addition | Points | On-call surface | Upgrade cadence | Expertise and debugging cost |
|---|---|---|---|---|
| Virtual machines | +1 | Instance failure, disk, patching | OS patches, image rebuilds | Widely understood; failures are local and legible |
| Multiple databases (polyglot) | +3 | Each engine fails its own way | Independent version cycles | Two consistency models, two backup and restore procedures, two sets of tuning knobs |
| Custom networking | +3 | Routing, address overlap, MTU, DNS | Rare but high-impact changes | Failures present as unexplained latency or intermittent timeouts, not as errors |
| Service mesh | +4 | Sidecars, certificates, control plane | Frequent, and in the request path | Every request now has a hop you must learn to read; certificate expiry becomes an outage class |
| Kubernetes | +5 | Control plane, nodes, scheduling, pods | Regular cluster and API upgrades with deprecations | A distributed system in its own right; new failure vocabulary before you can debug the application |
| Multi-region | +5 | Two of everything, plus replication and failover | Coordinated across regions | Data consistency becomes a design problem; the failover path is code nobody exercises |
What each addition actually costs you
The points are shorthand for four concrete burdens. On-call surface: every component is a new thing that can page someone at 03:00, and a new runbook that must exist and be accurate. Upgrade cadence: a managed orchestrator has supported-version windows measured in months, so you have signed up to a recurring upgrade project whether or not your application changes. Required expertise: not just to build it, but for whoever is on call to debug it under pressure, which means at least two people, which means hiring or training. Debugging difficulty: this is the one people underestimate. Each layer adds a place a request can be delayed or dropped, and a mesh in particular means a timeout might come from the application, the sidecar, the control plane's configuration or the certificate rotation — four hypotheses where there was one.
These compound rather than add. Kubernetes plus a mesh plus multi-region is not three problems; it is a cluster upgrade that must be coordinated across regions while the mesh control plane version is compatible with both, and a failure in that sequence affects the request path of every service. The score adds because the arithmetic must stay simple, but the lived experience is worse than linear — which is exactly why a team that is already at 14 should be much more reluctant to add the next component than a team at 3.
The counter-argument deserves fair hearing: complexity that is *required* is not waste. A platform serving regulated financial traffic across two continents with a 99.99% commitment genuinely needs several of these, and skipping them is not simplicity but negligence. The test is never "is this complex?" — it is "which requirement forces this, and what happens if we remove it?" A component with no answer to that question is the one to remove.
workload: internal dashboard, ~500 users, read-heavy team: 3 engineers (part-time on this) availability requirement: "a few minutes of downtime is fine" infrastructure: kubernetes: +5 # 2 clusters, quarterly upgrades multi_region: +5 # active-passive, failover never tested service_mesh: +4 # mTLS between 4 services polyglot_data: +3 # postgres + document store custom_networking: +3 # peering, custom routes kafka: +3 # counted as an extra data system redis_cluster: +2 # clustered, for a cache ---------------------- TOTAL: 25 requirement forcing multi-region: none stated requirement forcing the mesh: none stated requirement forcing kafka: none stated
workload: internal dashboard, ~500 users, read-heavy team: 3 engineers (part-time on this) availability requirement: "a few minutes of downtime is fine" infrastructure: managed container service: +1 # no control plane to upgrade managed relational db: +1 # backups and patching included multi_zone (single region):+1 # survives a zone loss cdn + object storage: +1 # static assets, no servers ---------------------------- TOTAL: 4 removed: second region, mesh, kafka, redis cluster, document store cache: in-process, 60s TTL — 500 users do not need a cluster async: the database table used as a job queue; revisit past ~50 jobs/s
Nothing in the first design was wrong in the abstract; every component is correct for some workload. None of them was forced by this one. The second design meets the same stated requirement with a quarter of the on-call surface, no recurring cluster upgrade, and failures that three part-time engineers can actually diagnose. If the requirement changes — a real availability commitment, real throughput — components come back one at a time, each with the problem that forced it.
The simplicity challenge: what can be removed?
Run this as a standing exercise rather than a one-off. Take the current architecture, list every component, and for each one write the single concrete requirement that forces it to exist. Not "scalability" — a number, an obligation, or an incident. If the sentence cannot be written, mark the component as a removal candidate. Then ask what specifically breaks if it is removed, and how you would know.
The answers are frequently uncomfortable. The message broker exists because a design document from 2023 said "event-driven", and it carries four messages a minute that a database table would carry perfectly well. The Redis cluster caches a query that takes eleven milliseconds. The second region has never served traffic and its failover has never been tested, which means it is not availability — it is a duplicate estate plus a belief. The service mesh provides mutual TLS between four services that could have had it from a load balancer.
Removing infrastructure is harder than adding it, socially and technically, so make the direction of travel explicit: new components require a stated requirement at review time, and existing components get periodically re-justified. And attach the score to the design review, alongside the cost estimate. A proposal that adds eight points to a team of three should have to say so out loud — not because it is forbidden, but because the person approving it should be approving the ongoing operational commitment, not just the architecture diagram. See Workload-First Thinking and Kubernetes Is Not Always Needed.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- A crude complexity score turns an argument about feelings into a comparison with units — its value is the conversation, not the number.
- The weights are simulated teaching values, not an industry constant; adjust them for what the team already operates fluently.
- Points stand for four real burdens: on-call surface, upgrade cadence, required expertise and debugging difficulty.
- Complexity compounds rather than adds — a cluster upgrade coordinated across regions with a mesh in the path is worse than the sum of its parts.
- Required complexity is not waste. The test is which requirement forces each component, and what breaks if it is removed.
- Score the design you have against the simplest design that meets the requirement; the gap is the thing to explain.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Enumerate every infrastructure component in the design, including the ones inherited rather than chosen.
- • Assign each a weight reflecting the ongoing operational surface it adds for this team, not its inherent sophistication.
- • Sum the weights, and record the total alongside the team size and the workload's stated availability and throughput requirements.
- • Independently score the simplest architecture that satisfies those same requirements.
- • For each point of difference, name the requirement that forces it; components with no named requirement become removal candidates.
- • Re-score at every significant design change, so the trend is visible rather than the snapshot.
- • You own a runbook per component, kept accurate — an out-of-date runbook is worse than none because it is trusted at 03:00.
- • You own an upgrade calendar for every platform with a supported-version window, and the project time it consumes.
- • You own redundancy in people: at least two who can debug each technology, or the score understates your risk considerably.
- • You own periodic re-justification of existing components, because removal never happens without a scheduled prompt.
- • You own the score itself as a design-review artifact, presented next to the cost estimate.
- • Complexity exceeding team capacity: incidents last longer, one person becomes the only one who understands the platform, and they eventually leave.
- • Upgrade debt accumulating until a platform falls out of support and the upgrade becomes a migration.
- • An incident with four plausible layers to blame and no way to distinguish them quickly, turning a ten-minute outage into a ninety-minute one.
- • Untested failover paths in a multi-region design providing the appearance of availability and none of the substance.
- • The score used as a target rather than a conversation, so people argue about weights instead of about requirements.
- • Complexity justified by an anticipated future scale that never arrives, paid for monthly in the meantime.
- • Operational load scales with component count and team size together — the same architecture is sustainable for twenty engineers and unsustainable for three.
- • Debugging time scales with the number of layers in the request path, which is why a mesh costs more than its resource footprint suggests.
- • Upgrade burden scales with the number of platforms that have version windows, independently of traffic.
- • The dimension that runs out first is almost always human attention, not capacity.
- • Every component is an additional attack surface, an additional set of credentials and an additional patching obligation.
- • Complexity degrades security review quality: nobody can hold a twenty-five-point architecture in their head well enough to reason about its blast radius.
- • A service mesh genuinely improves the security posture — mutual TLS between services is real — which is why it is a legitimate +4 rather than pure overhead, and why the question is whether that requirement exists here.
- • Unused components are the least patched, because attention follows traffic; the retired-but-still-running service is a standard breach path.
- • Simpler architectures have smaller and more comprehensible blast radii, which is a security benefit rarely counted as one.
- • Direct infrastructure spend is usually the smallest cost of complexity, which is why cost reviews alone never catch over-engineering.
- • Engineering time — upgrades, incidents, onboarding — is the largest, and it is charged to salary rather than to the cloud bill.
- • Lost change velocity appears as a slow product roadmap and is almost never attributed to the infrastructure that caused it.
- • Idle duplicated capacity in unjustified multi-region designs is a real and easily measured line item. See Idle Capacity: Headroom or Waste?.
- • On-call burden has a turnover cost, which is the most expensive item on this list and the one no budget contains.
- • Incident count and mean time to diagnose, split by component — the empirical version of this score.
- • Time spent on platform upgrades per quarter, as a fraction of engineering capacity.
- • Bus factor per technology: how many people could debug this under pressure at 03:00.
- • Onboarding time to first production change, which is the most honest measure of how complex a system really is.
- • The signal that lies: a green dashboard on a system nobody understands. Availability metrics say nothing about whether the next incident is diagnosable.
- • The simplest architecture that meets the stated requirement, which for a great many internal systems is a managed container service, a managed database and multi-zone within one region.
- • A modular monolith on virtual machines. Fewer moving parts, one deploy, one log stream, and entirely sufficient at most real traffic levels.
- • A managed platform instead of an orchestrator you run yourself — same workload, several points removed from the score.
- • Adding components one at a time, each triggered by an observed problem rather than an anticipated one. This is the whole argument of the laptop-to-production progression: every component must name the problem that forced it. See From Laptop to Production.
- • Removing something. The most under-used option in infrastructure, and the only one that reduces the score.
- • A score buys a shared vocabulary for a previously unwinnable argument; costs precision, and invites debate about the weights rather than the requirements.
- • Simplicity buys diagnosability and team capacity; costs headroom for growth you may genuinely need later.
- • Each component buys a capability and costs on-call surface, an upgrade cadence and an expertise requirement — permanently, not once.
- • Removing complexity buys back capacity; costs migration work and the political difficulty of undoing someone's earlier decision.
Operational complexity, scored
Do you need Kubernetes?
Where the bill actually comes from
fixed weight is committed at provision time; usage weight follows the workload. idle = 100% − 35% used → headroom 25% (chosen) + waste 40% (not chosen)
What people believe, and what is true
A lower score is always better.
The score is meaningless without the requirement next to it. A payments platform at 20 may be correctly built; an internal dashboard at 20 is not. The comparison is against the requirement, never against another team.
We will need this complexity eventually, so we should build it now.
You pay the operational cost every month between now and then, you build it against guessed requirements, and most systems never reach the scale that was assumed. Add components when a problem forces them.
Kubernetes reduces operational complexity because it automates operations.
It automates container placement and restarts, and adds a control plane, a scheduler, an upgrade cadence, a new failure vocabulary and an expertise requirement. For a small workload the trade is clearly negative; at scale it clearly reverses.