Declarative vs Imperative Infrastructure
Describing the end state and letting a tool derive the steps, versus writing the steps yourself — and the cases where writing the steps is still correct.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Is describing the desired end state actually better than scripting the steps, or is that just fashion?
A script that creates infrastructure works once. Run it again and it either fails because things exist, or creates a second copy. Infrastructure changes are not one-off events, so a one-off program is the wrong shape.
Write a shell script against the provider CLI. It is readable, you can run it line by line, and there is no state file, no lock, no provider plugin and no new language to learn.
The second run is a different program from the first. Making it idempotent means hand-writing "does it exist, is it correct, change only what differs" for every resource — which is the tool you were avoiding, written worse.
- The second run is a different program from the first. Making it idempotent means hand-writing "does it exist, is it correct, change only what differs" for every resource — which is the tool you were avoiding, written worse.
- It has no dry run. You find out what the script does by letting it do it.
- Deleting a resource means remembering to delete it from the script and writing a delete step, because removing creation code deletes nothing. Resources become permanent by accident.
- Failure halfway leaves the system in a state the script cannot reason about. Re-running compounds it.
- Ordering is implicit in line order, so the dependency graph lives in the author's head and is discovered by the person who reorders two blocks.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Declarative: you write the end state. The tool reads current state, computes a diff, orders the operations from the dependency graph, and executes. Deleting a block is a real instruction — it means "this should not exist" — which is exactly why a rename is dangerous (Destructive Changes: What a Rename Really Does).
- Imperative: you write the transition. The tool executes it in order and has no model of the end state, so it can neither dry-run nor detect drift nor derive a delete.
- The dividing line is who computes the diff. Declarative tools need a way to know current state, which is either a recorded state file, a server-side stack, or a live read of every resource. That requirement is the price of the dry run.
- Neither is a property of the language. HCL is not what makes Terraform declarative, and Pulumi programs written in TypeScript are declarative because they build a resource graph rather than executing calls in order.
- Almost every real system is a hybrid: declarative for the infrastructure graph, imperative for the operations that are genuinely sequences — a data migration, a failover, a one-time backfill.
The same resource, two ways
The imperative version is shorter and more obvious on the first run. Compare what happens on the second, and on the day someone deletes the resource from the file.
create bucket assets enable versioning on assets set lifecycle rule on assets # second run: fails, "already exists" # fix: add "if not exists" to every line # delete the lines: nothing is deleted # dry run: none
bucket "assets" {
versioning = enabled
lifecycle = 90 days
}
# second run: no changes
# delete the block: plan shows destroy
# dry run: the plan, before anything happensThe imperative version encodes the transition from one specific starting point. The declarative version encodes the destination, so it is valid from any starting point — and that is what makes both re-running and deleting meaningful operations rather than accidents.
Choosing per operation, not per team
This is not a stack decision made once. It is a per-operation decision, and mature setups use both deliberately.
You need to make an infrastructure change. Declarative or imperative?
when The thing has a lifecycle: it will be changed, replicated across environments, or needs to be reviewable before it happens.
cost State to manage, a plan to read, and provider-specific replacement rules to learn.
when A genuinely ordered one-off: a cutover, a failover, a data backfill, a manual verification between steps.
cost No dry run, no drift detection, and the rollback must be written separately and rehearsed.
when The provider has no API for the thing and you have no alternative.
cost You lose idempotency exactly where the tool assumes it. Every retry is a second execution.
when The platform already runs a control loop that reconciles continuously, as Kubernetes does.
cost Changes are eventually consistent rather than transactional, so "applied" and "in effect" are different moments (Apply Is Not Running).
Ordering is derived, not written
The most common surprise when moving from scripts to declarative tools: line order stops mattering, and reference order starts. A resource is created after the things it references because the tool built a graph, not because it appears later in the file.
This is a real improvement — the graph is correct even when the file is reordered — and it becomes a problem when a dependency is real but not expressed as a reference. The classic case is a permission that must exist before a service starts using it, where nothing in the configuration references the permission.
- 1Parse
Read configuration and expand modules, counts and loops into concrete resource addresses.
fails by A loop key derived from a value that is not known until apply, so the tool cannot expand the graph at plan time.
evidence The plan lists every address explicitly rather than deferring.
- 2Build the graph
Draw an edge wherever one resource references another's attribute.
fails by A real ordering requirement that is not a reference, producing a race the tool cannot see.
evidence Applies succeed from an empty state, not just incrementally.
- 3Order and parallelise
Execute independent branches concurrently, dependents after dependencies.
fails by Provider rate limits under wide parallelism, which surfaces as intermittent apply failures.
evidence Repeated applies from empty succeed without retries.
- 4Record
Write each created resource's real id back to state as it completes.
fails by Interrupted apply, leaving some resources created and some unrecorded (State).
evidence A follow-up plan is empty rather than offering to create things that already exist.
How to do it properly
Most important first.
- Use declarative for anything with a lifecycle: things that will be changed, replaced, or need to exist in more than one environment.
- Use imperative for genuinely ordered operations that happen once — a cutover, a region failover procedure, a backfill (Backfills, Region Failover).
- When you must reach for imperative inside a declarative tool — a provisioning script, a local command — treat it as a code smell you have chosen deliberately, and make it idempotent, because the tool will not.
- Do not fight the model. Encoding a sequence into a declarative tool with artificial dependencies produces a graph nobody can read and a plan nobody can predict.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Choice of model is contained by environment separation; a script with production credentials and no dry run is contained by nothing.
What can go wrong
- Declarative configuration wrapped in a shell script that runs it with different variables per environment, reintroducing every problem the tool removed.
- Provisioning steps embedded in the resource graph, so an unrelated change re-runs a script against a live machine (Immutable Infrastructure is the alternative).
- Imperative escape hatches that are not idempotent, so a retried apply does the operation twice.
- A declarative model of something the provider does not expose declaratively, held together by manual imports.
- "Declarative means idempotent." It means the tool computes the diff. Provisioners, local commands and provider bugs can all make an apply non-idempotent.
- "Imperative is legacy." A failover procedure is inherently ordered. Writing it declaratively does not make it better, it makes it obscure.
- "YAML is declarative and code is imperative." The distinction is whether the tool derives the transition, not what the input file looks like (Declarative vs Imperative Infrastructure in Cloud & Infrastructure makes the same point about provisioning APIs).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Running the same configuration twice with no source change produces no operations. That single fact is the whole difference.
- A reviewer can predict the effect of a diff without running anything, for additive changes at least.
- Deleting a block produces a plan that deletes the resource — the delete path exists and was not forgotten.
- Declarative rollback is "restore the previous configuration and apply", which is genuine for additive changes and a restore-from-backup problem for anything stateful.
- Imperative rollback has to be written by hand as a second script, and it is written under time pressure, which is when it is least likely to be correct.
- Automate the declarative path completely: plan, policy check, apply.
- Keep the imperative operations attached to a runbook with a human running them, because their whole nature is that they are ordered, one-off, and context-dependent (Runbooks).
- The dry run is paid for with state. Everything painful in State is the cost of being able to see a plan.
- Declarative tools are worse at expressing "do A, wait for a human, then do B". That is a workflow, and it belongs in a workflow tool.
- You inherit the provider maintainer's model of each resource, including their choice of which attributes force replacement.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe diff-computed-by-the-tool distinction holds across Terraform, CloudFormation, Pulumi, Kubernetes manifests and Ansible in its idempotent-module form. What differs is where current state comes from.
- TOOL-SPECIFICTerraform derives ordering from references between resources and can be forced with explicit dependencies. Kubernetes derives nothing and relies on controllers converging repeatedly instead, which is why an ordering bug there is eventually consistent rather than fatal (Reconciliation: The Loop Under Everything).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — convergence and reconciliation as an alternative to transactional application of a desired state.