State
The mapping between configuration addresses and real resources — why it must exist, why it goes stale, why it holds secrets, and what concurrent applies do to it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does the tool need a state file at all when it could just ask the cloud what exists?
The provider can tell you what resources exist. It cannot tell you which of them corresponds to the block of configuration you just edited, or that a resource you deleted from the file was ever yours to delete.
Keep the state file next to the code, commit it to git so everyone has it, and get on with the work.
Two people apply at the same time from two checkouts. Both write state. One set of resources is now orphaned — real, costing money, and invisible to the tool.
- Two people apply at the same time from two checkouts. Both write state. One set of resources is now orphaned — real, costing money, and invisible to the tool.
- The state file contains resource attributes verbatim, including generated passwords and private keys. Committing it puts secrets in git history, where they stay after the file is deleted (What Counts as a Secret, and Where It Must Not Be).
- Merge conflicts in state are not resolvable by reading them. A hand-merged state file is a fabricated claim about reality.
- Someone works from a stale checkout, applies, and the tool creates a second copy of resources it does not know about.
- The laptop with the only current state dies, and the tool now believes production does not exist.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- State is three things: a map from configuration address to real resource id, a cache of each resource's attributes as last seen, and a record of dependencies so destroy can run in the correct order.
- The map is the part that cannot be reconstructed from the provider. Ten identical subnets exist; which one is
module.network.subnet[2]is not knowable from the cloud API alone. - The attribute cache is what makes refresh a comparison rather than a full read, and it is what goes stale when someone changes something outside the tool.
- Locking is a separate mechanism layered on the backend: acquire an exclusive lock, refresh, plan, apply, write, release. Without it, two concurrent applies interleave their writes and the last writer wins with a partial picture.
- Sensitive values are in state because the tool stores what the provider returned. Marking an output sensitive hides it from the console output; it does not remove it from the file. The file is the secret.
What state stores, and which part cannot be rebuilt
Three distinct jobs are bundled into one file, and they have very different recovery stories. Knowing which is which tells you how bad a given state incident is.
| What it holds | Why the tool needs it | If you lose it |
|---|---|---|
| Address to real id | To know that db.primary in the code is that specific instance | Not recoverable automatically — every resource must be imported by hand |
| Cached attributes | To diff without a full read, and to detect drift | Recoverable: a refresh repopulates it |
| Dependency order | To create and, in reverse, destroy in a valid order | Recoverable from configuration, unless the configuration is also gone |
| Whatever the provider returned | Nothing — it is a side effect of storing attributes | This is where generated passwords and keys live (Secrets in CI) |
Backend and lock
Remote state with locking is not an optimisation for large teams. It is the mechanism that stops two runs from writing incompatible pictures of reality, and one team with one pipeline can trigger that on any re-run.
1terraform {2 backend "s3" {3 bucket = "acme-tfstate-prod"4 key = "platform/network/terraform.tfstate"5 region = "eu-west-1"6 dynamodb_table = "acme-tfstate-locks"7 encrypt = true8 }9}The key is the blast-radius decision: one state per environment and per component, not one key for everything. The lock table is a separate resource, which means it lives in its own state — bootstrap it once and leave it alone.
State incidents, and the response that does not make it worse
Every row here has a wrong response that is faster and a right response that is slower. The wrong one usually involves editing the file.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Run crashed mid-apply | Lock held, every subsequent run blocks | The lock is released on clean exit, and the process did not exit cleanly | Confirm the run is genuinely dead — check the pipeline, not the clock — then force-unlock. Forcing a live run corrupts state |
| Resource deleted in the console | Plan wants to update something that no longer exists; apply errors | The cached attributes describe a resource that is gone | Refresh so the tool learns it is gone, then let the plan recreate it — or remove it from state if it should stay gone |
| Module refactor, resources moved | Plan destroys and recreates everything that moved | A new address with no state entry looks exactly like a new resource (Destructive Changes: What a Rename Really Does) | Use moved blocks in configuration, or terraform state mv, before applying |
| State restored from an older version | Plan wants to create resources that already exist | The restored map predates those resources | Import them rather than applying. Applying creates duplicates and orphans the originals |
| Two states include the same resource | Applies flap the resource back and forth | A split that copied entries instead of moving them | Remove from one state — remove, not destroy — and confirm with an empty plan in both |
| State bucket unreadable | No infrastructure change is possible | Permissions change, or the backend is in the region that is currently failing | Keep the backend outside the failure domain it describes; this is a dependency in your recovery plan (Disaster Recovery as an Operation) |
How to do it properly
Most important first.
- Remote backend, always, with locking and versioning enabled. Version history on the state object is the only realistic recovery path when state is damaged.
- Encrypt the state at rest and restrict read access to the pipeline identity plus a small break-glass group. Treat read access to state as equivalent to read access to the secrets it contains (Secret Managers and What They Actually Give You).
- One state per environment, and separate state for resources with different blast radius. Shared state means every apply holds a lock every other change waits on.
- Never edit state by hand. Use the tool's own state operations — move, remove, import — which keep the internal structure consistent.
- Where possible, do not put generated secrets in state at all: let the provider manage the credential and reference it, or generate it in a secret manager and read it as a data source (Workload Identity).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Per-environment state, locking, and versioned backends. Corrupted state in a single shared global state is contained by nothing but the destroy gate on the next plan.
What can go wrong
- Lock left held by a crashed run, blocking all applies until someone force-unlocks — and force-unlocking a run that is actually still in flight is worse than waiting.
- State and reality diverge because someone deleted a resource in the console: the tool now plans to update something that no longer exists.
- Two states owning the same resource, usually after a refactor split a module. Both applies fight, and the resource flaps.
- A refactor that moved resources between modules, applied without moving them in state, producing a plan that destroys and recreates everything it moved.
- State backend in the same account and region as the infrastructure it describes, so a regional failure takes the recovery tool with it.
- "State is a cache, so losing it is recoverable." The attribute cache is recoverable; the address-to-id map is not, and rebuilding it means importing every resource by hand.
- "Sensitive markings protect secrets in state." They control display, not storage. If a secret was returned by the provider, it is in the file in plaintext.
- "We do not need locking, we are a small team." Locking protects against your own pipeline running twice, which happens the first time someone re-runs a stuck job.
- "CloudFormation has no state." It has state; the state is server-side and you cannot lose it or corrupt it locally. That removes this whole class of problem and replaces it with drift you cannot repair as directly.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The state object has versioning on, and a previous version has actually been restored in a drill.
- Concurrent pipeline runs serialise on the lock rather than racing — visible as one run waiting, not two succeeding.
- Access logs on the state bucket show reads from the pipeline identity and named humans only (Access Review).
- Restoring a previous state version rolls back the tool's belief, not the infrastructure. After a restore the two disagree, and the next plan will offer to fix the difference — read that plan extremely carefully.
- For a partially applied run, the state is usually correct: it records what was actually created. Re-running is normally right; restoring an older state is normally wrong.
- Automate: backend configuration, locking, encryption, versioning, and access via a pipeline identity rather than user credentials.
- Keep human: force-unlock, state surgery, and any decision to restore an older state version. Each of these is a claim about reality that only a person can verify (Break-Glass Access).
- Splitting state reduces blast radius and lock contention, and creates cross-state references that are harder to reason about and can go stale in their own way.
- Remote state adds a dependency on the backend being available. Applying during an incident is impossible if the backend is in the failing region.
- Locking serialises infrastructure changes, which is correct and slow. Large organisations feel this as a queue.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICThis is Terraform/OpenTofu state semantics: an explicit file you own, with a pluggable backend and an advisory lock. Pulumi keeps equivalent state in a service or self-managed backend and locks per stack. CloudFormation holds state server-side per stack, so there is no file to lose, no lock to force-unlock, and no state surgery — which also means there is no way to tell it that a resource moved.
- CLOUD-SPECIFICThe locking mechanism follows the backend: an object store plus a conditional-write or lock table on one provider, a blob lease on another. Their failure modes differ — a lease expires on its own, a lock row does not.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — mutual exclusion over a shared object store, and why an advisory lock plus a version check is not the same as a transaction.