The question this answers
How do I make the infrastructure that runs production reproducible, reviewable and recoverable instead of a shape that exists only inside one account?
The payments stack must be reproducible: a second region stood up in a day, an accidentally deleted subnet restored without archaeology, and an auditor answered when they ask who opened the database port to the internet in March and which ticket authorised it.
A declarative definition of every resource in version control, so an infrastructure change gets the same pull request, review, history, revert and CI path as an application change.
The console is a write-only interface
Clicking through a provider console produces working infrastructure and no record of how it was produced. The account holds the *result* — a security group with four rules — and nothing about the reasoning, the author, the date or the two rules that were added during an incident and never removed. Six months later the only way to answer "why is this here?" is to ask whoever is still on the team, and the honest answer is usually "it was already like that".
Infrastructure as code inverts that. The file is the source of truth and the account is a *projection* of it. The tool reads the definition, reads what actually exists, computes the difference, and makes reality match. That single inversion is what buys everything else: a diff in a pull request, a git blame on a firewall rule, a revert that is a revert rather than a memory exercise, and a second environment that is a variable change instead of a two-week project.
It also changes who is allowed to change production. Once the pipeline holds the credentials, human write access to the console can be reduced to break-glass — which is a security win the reproducibility argument tends to get the credit for. See Roles vs Static Keys and Audit Trails.
- 1Infrastructure definition
Resources, their arguments and their relationships, written as files in a repository and reviewed like code.
The definition drifts from intent when changes are made in the console instead — the file still says what someone meant last quarter.
- 2Planseconds to a few minutes — ILLUSTRATIVE
The tool reads the definition, reads recorded state, queries the provider, and prints exactly what it would create, change or destroy.
A plan nobody reads. The
-/+replace lines are the ones that matter and they scroll past in CI logs. - 3Apply
The tool calls the provider APIs in dependency order and records the resulting resource identities.
A partial apply: half the graph created, then a quota error. Reality is now a state neither the file nor the previous state describes.
- 4Cloud resources
Real, billed, reachable infrastructure whose identity is recorded so the next plan can compare against it.
Someone edits it by hand. See Drift: When the File and Reality Disagree.
What it buys, measured against the honest alternatives
The interesting comparison is not IaC against clicking. It is IaC against a shell script that calls the provider CLI, which is what most teams actually have. A script is versioned and reviewable too. What it cannot do is tell you what it is *about* to change, or notice that reality has moved, because it has no model of desired state — it only knows the operations it is about to perform.
The cost side is real and usually understated: a language to learn, a state store to run and secure, a plan/apply pipeline with production credentials, and a new class of outage where the tool is the thing that broke. For a two-person team running one server, a documented README with the five commands is a legitimate answer and IaC is overhead. The threshold is roughly the point where you have a second environment, or a second person, or a compliance question.
| Approach | Reproducible? | Reviewable diff? | Detects drift? | Honest use case |
|---|---|---|---|---|
| Console clicks | No — only in someone's memory | No | No | A throwaway experiment you will delete today. |
| Documented runbook | By hand, with transcription errors | The prose changes, not the effect | No | One server, one person, no compliance obligation. |
| Provider CLI script | Yes, if it is idempotent — most are not | You review operations, not outcomes | No | A one-off operational task: rotate a key, drain a node. |
| Declarative IaC | Yes, from the file | Yes — the plan is the diff | Yes, that is what a plan is | Anything with more than one environment or more than one owner. |
The definition should describe the outcome, not the ritual
The most common way IaC adoption fails is that the team keeps writing scripts, just in a new syntax: a chain of resources with hardcoded ids, one file per environment, copy-pasted and edited. It is versioned, so it feels like progress, but the second environment is still a manual transcription and the diff still cannot be read.
The version on the right below is not longer or cleverer. It names the relationship between resources instead of the value that happens to satisfy it today, which is what lets the tool build a dependency graph, order the API calls correctly and rebuild the whole thing in an empty account.
resource "cloud_instance" "api" {
subnet_id = "subnet-0a91f3c7" # copied from the console, prod only
image_id = "img-2024-06-11-hotfix"
}
resource "cloud_security_rule" "db" {
source_group = "sg-4471bc02" # which one is this? nobody knows
port = 5432
}resource "cloud_instance" "api" {
subnet_id = cloud_subnet.private_a.id
image_id = var.api_image_id
}
resource "cloud_security_rule" "db" {
source_group = cloud_security_group.api.id
port = 5432
description = "API tier to Postgres"
}References create edges in the dependency graph. Hardcoded ids create a file that only applies cleanly in the one account those ids came from, and a plan that cannot explain what a change will cascade into.
Key points
- IaC inverts the source of truth: the file describes the intended infrastructure and the account becomes a projection of it.
- The reviewable diff, not the automation, is the main benefit — a firewall change becomes a pull request with an author and a reason.
- A provider CLI script is versioned but has no model of desired state, so it can neither preview a change nor detect drift.
- The real costs are a state store to secure, a pipeline holding production credentials, and a new outage class where the tool is what broke.
- One server and one engineer with no second environment is a legitimate case for a documented runbook instead.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Resources are declared as typed blocks with arguments; references between them (
cloud_subnet.private_a.id) form a directed dependency graph. - • The tool loads recorded state — its map from declared resources to real provider identities — and refreshes it against the provider API.
- • It diffs desired configuration against refreshed state and emits a plan: create, update in place, replace, or destroy, per resource.
- • On apply it walks the graph in dependency order, calling provider APIs, and writes the resulting identities back into state.
- • The next run repeats the comparison, which is why running it twice with no change is a no-op — the defining property of a declarative tool.
- • You own the state backend: where it lives, who can read it, how it is locked, and how it is backed up. See State: The File That Makes It Work and the File That Will Hurt You.
- • You own provider version pinning. An unpinned provider upgrade can turn a no-op plan into a destroy-and-recreate on a resource nobody touched.
- • You own the pipeline identity that applies changes, which by construction has broad production write access.
- • You own the modules, their interfaces, and the migration when an interface changes under twenty callers.
- • You still own everything the resources themselves need: schemas, capacity choices, patching, and the bill.
- • A partial apply after a quota or permission error: some resources created, state written, and a graph that matches neither the file nor the last known good.
- • A provider upgrade changes a default, and the next plan proposes to replace a database because an attribute the team never set is now computed differently.
- • Someone applies from a laptop with a stale state file and reverts three weeks of infrastructure changes in one run.
- • A resource is deleted from the configuration; the plan destroys it; nobody notices that the resource was the one holding the data.
- • The tool itself becomes unavailable — locked state, expired credentials — during an incident, so the fastest fix is a console change that immediately becomes drift.
- • The thing that runs out first is plan time. A single state file holding a thousand resources refreshes every one of them against the provider API on every run, and a five-minute plan stops being read.
- • Split by blast radius and change frequency: network and identity change rarely and are catastrophic; application resources change hourly. They do not belong in one state.
- • Provider API rate limits become the ceiling on a large refresh, which shows up as intermittent plan failures rather than as slowness.
- • Team scaling breaks first through lock contention: one state, twenty engineers, and a queue of people waiting to apply.
- • The pipeline identity is the most privileged identity in the system — it can create, modify and destroy everything. Treat it as a crown-jewel credential, not as CI plumbing.
- • State contains resource attributes including, on many providers, generated passwords and keys in plaintext. Encrypt it, restrict read access, and audit access to it.
- • IaC is a security *win* when it removes standing human console write access and replaces it with reviewed, logged, revertible changes.
- • A pull request that changes a security group is a security review artifact. Require an approver on those paths specifically, not on the repository in general.
- • The tooling is usually cheap or free; the cost is engineering time to build modules and the pipeline, and a state backend that is a rounding error.
- • The real financial effect is indirect and large: reproducible environments make it trivial to create a full copy per developer, and that is exactly how a bill triples.
- • Destroy-on-schedule for non-production environments is only possible *because* they are code — the same property that made them easy to create makes them easy to reclaim.
- • Plan output on every pull request, and a plan on a schedule against the untouched main branch — a non-empty plan there is drift.
- • Apply duration and failure rate: a rising apply failure rate usually means provider version skew or a state that is too large.
- • Provider audit logs, correlated with the pipeline identity — a production change made by a human identity is an event, not a data point.
- • The signal that lies: a green pipeline. It says the apply succeeded, not that the resulting infrastructure is healthy; the health check belongs to the workload, not the tool.
- • A documented runbook and console clicks, for a single environment run by one or two people with no audit obligation. Genuinely correct at that size.
- • A provider CLI script for a one-off operational task — rotate a credential, drain a node, resize a volume once. Declarative tools are poor at operations that are inherently a sequence.
- • The provider's own stack service (CloudFormation, ARM/Bicep, Deployment Manager) when you are on one provider and want the state managed for you — you trade portability for one less thing to secure.
- • A platform-as-a-service that has no infrastructure to define. If the answer to "what infrastructure?" is "a container and a database", IaC may be solving a problem you do not have yet.
- • Buys reviewable, repeatable change; costs a new language, a state store to secure, and a pipeline with production write access.
- • Buys drift detection; costs a discipline that every change goes through the file, including at 3am during an incident when the console is faster.
- • Buys environment parity; costs the temptation to make every environment a full-scale copy, which is a bill problem. See Development, Staging and Production.
What people believe, and what is true
IaC means infrastructure is automated.
IaC means infrastructure is *defined*. Automation is what you build on top when a pipeline applies it. Plenty of teams run IaC entirely from laptops, and get most of the review benefit and none of the audit benefit.
Once we have IaC, nobody touches the console.
Someone will, during an incident, correctly. The discipline is not "never" — it is reconciling it back into the file the same week. See Drift: When the File and Reality Disagree.
IaC makes infrastructure changes safe.
It makes them visible. A reviewed plan that destroys the production database is still a destroyed production database. Visibility only helps if someone reads the plan.