Ephemeral Environments
Environments created on demand and destroyed when done, which is only possible once infrastructure, configuration, data and secrets are all codified — and which turns environment count into a decision instead of an inheritance.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What has to be true before an environment can be created and destroyed on demand rather than kept running forever?
Long-lived environments are expensive, drift continuously, and are contended — but recreating one is impossible when nobody knows all the manual steps that produced the current one.
We will make environments ephemeral by writing a Terraform module for them. Once the infrastructure is codified, we can create and destroy at will.
Infrastructure is the easy half. An environment is only useful once it has configuration, seed data, secrets, DNS, certificates and registered third-party callbacks — and those are usually the parts that were done by hand.
- Infrastructure is the easy half. An environment is only useful once it has configuration, seed data, secrets, DNS, certificates and registered third-party callbacks — and those are usually the parts that were done by hand.
- Creation time becomes the binding constraint. If a full environment takes forty minutes, people keep one running instead, and you are back where you started with worse tooling.
- Data is the hard problem. Restoring a realistic dataset quickly is a genuine engineering effort, and doing it by copying production creates an exposure (Production Data in Lower Environments).
- Third-party integrations often cannot be created on demand: a sandbox account, a webhook registration, an OAuth callback allowlist or a payment provider account may be a manual, rate-limited or contractual step.
- Destroying reliably is harder than creating. Orphaned load balancers, volumes, DNS records and cloud roles are the residue, and they cost money and hold permissions (Idle Capacity).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An ephemeral environment is the cattle model applied to a whole stack: it is defined entirely by code plus data inputs, so any instance is reproducible and no instance is precious (Pets and Cattle, Read Carefully).
- That forces every manual step into the open. The first serious attempt at teardown-and-recreate is a discovery exercise: everything that fails is a step nobody had written down.
- Once it works, two properties follow. Drift becomes bounded, because instances do not live long enough to accumulate it. And environment count becomes elastic — you create one for a rehearsal and destroy it, instead of budgeting for a permanent one.
- Lifecycle needs an owner: something must destroy environments whose purpose has ended, or the cost model collapses back into the long-lived case with extra steps.
The five things that must be codified, in the order they block you
Every team that attempts this discovers the same ordering: infrastructure is codified first because it is the visible part, and then the project stalls on the four items below it.
| Layer | Codified how | What stalls the project |
|---|---|---|
| Infrastructure | Declarative definitions, versioned, reviewed (Infrastructure as Code) | Usually already done — this is the easy half |
| Configuration | Per-environment values derived from one schema (Artifact Plus Configuration) | Keys that only exist in one environment and nobody remembers adding |
| Secret access | Identity-based issuance at boot; never values in the definition (Workload Identity) | A long-lived key pasted into a console two years ago |
| Data | Generated synthetic seed, or restore from an anonymised snapshot | The hardest item, and the one most often solved badly by copying production |
| Network identity | DNS records and certificates issued per environment (Renewal: Automating the Thing That Expires) | Wildcard certificates and manual DNS entries that do not scale to N environments |
| Third-party registration | Pooled pre-registered sandbox accounts, borrowed and returned | Accounts that require a human, a contract or a rate-limited signup |
Long-lived or on demand
This is not a one-way door, and both models are legitimate. The failure is choosing on-demand for the aesthetics and then keeping a long-lived environment anyway because the on-demand path is not reliable enough to trust.
env-a .. env-e, always running
each drifts independently
each holds its own credentials
each billed 24/7, used ~6h/week
nobody owns c and d
recreating any of them: unknown,
never attemptedone definition in the repository
create on demand, ttl 24h
reaper destroys on expiry
creds minted at boot, revoked on teardown
billed while used
recreation: exercised daily,
so it is known to workThe right column is not cheaper because it runs less — that is a secondary effect. It is safer because recreation is exercised constantly, so the recreation path is known to work at the moment you need it, which is usually during a recovery (Restore Drills). The left column's recreation path has never been run and is a guess.
The reaper, and why it is the hard part
Creation is a happy path with an obvious owner. Destruction has no owner, no urgency, and fails quietly — which is why the cost model of ephemeral environments collapses in practice.
The failure table below is what actually goes wrong once a team is past the first month.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Time-to-live expires mid-use | Environment disappears during a demo or a rehearsal | A fixed TTL with no extension path and no warning | Warn before expiry, allow extension by the owner, require a reason for long extensions |
| Teardown partially fails | Bill does not drop after environments are destroyed | Retained-by-default resources — disks, snapshots, addresses, log storage — survive their parent | Reconcile a tag-filtered inventory against the list of live environments, not against teardown exit codes |
| Environment destroyed, credentials not revoked | Live credentials exist for a stack that no longer runs | Credential issuance is automated and revocation is not (Rotation That Applications Survive) | Bind credential lifetime to the environment; prefer short-lived tokens that expire without action (Short-Lived Credentials) |
| Reaper deletes by name prefix | A shared or production resource is destroyed | Selection by naming convention rather than by an owner tag, and one resource was named badly | Select by tag, require the tag at creation, and refuse to delete anything untagged |
| Creation intermittently fails | People keep a long-lived environment "just in case" | A dependency with a rate limit or a manual step still in the path | Treat creation reliability as a service level; the whole model rests on it |
| Seed data grows stale | Environments are created successfully and are useless | The seed script was written once and never revisited as schemas changed | Seeding runs against the current schema in CI, so a stale seed fails a build rather than a person |
How to do it properly
Most important first.
- Codify in this order — infrastructure, then configuration, then secret *access* (never secret values), then data seeding, then DNS and certificates. The later items are the ones that block teardown.
- Issue secrets through workload identity at boot rather than baking them into the environment definition, so a new environment gets credentials without anyone copying a value (Workload Identity).
- Attach a time-to-live to every environment at creation and have a reaper enforce it. Opt-in cleanup does not happen.
- Tag every resource with the environment identifier and owner so orphans are findable and deletable (Cost Engineering).
- Optimise creation time deliberately: pre-baked images, cached artifacts, and a seed dataset that restores from a snapshot rather than running through the application (Caching in CI).
- Handle un-createable dependencies explicitly with a small pool of pre-registered sandbox accounts that environments borrow and return.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained to the environment being created or destroyed, provided isolation is real. Two exceptions escape it: teardown that deletes a shared resource because a tag was wrong, and credentials outliving the environment they were minted for.
What can go wrong
- The reaper deleting an environment someone was mid-way through using, because time-to-live had no extension mechanism.
- Teardown that leaves orphans — volumes, snapshots, load balancers, DNS records, cloud roles — which cost money and quietly retain permissions (Access Review).
- A creation path that succeeds ninety percent of the time, which is enough to be maddening and not enough to be trusted, so people keep a long-lived environment as insurance.
- Ephemeral environments that quietly all point at one shared database because seeding was too slow, removing the isolation that justified them.
- Secret sprawl: each new environment mints credentials that are never revoked on teardown, leaving live credentials for stacks that no longer exist (Rotation That Applications Survive).
- "Ephemeral environments mean we no longer have drift." They bound drift within one environment's lifetime. The *definition* still drifts from production, and now it drifts silently because no long-lived instance is around to make it visible.
- "If it is in Terraform, it is reproducible." State, data, secrets, DNS and third-party registrations usually are not, and those are what block recreation (State).
- "They are cheaper." They are cheaper only if teardown is reliable. Ephemeral environments with unreliable teardown are more expensive than permanent ones, because nobody is even counting them.
- "Everything should be ephemeral." Some environments genuinely should persist — a long-running performance baseline environment loses its value if recreated with a different dataset every time (Load Testing: What Question Is This Test Answering?).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- An environment can be destroyed and recreated from its definition, by someone who did not build it, with no manual steps and no help.
- Median creation time is measured and short enough that people choose creation over reuse.
- A resource inventory filtered by environment tag shows no resources belonging to environments that no longer exist.
- Credentials issued to a destroyed environment are demonstrably revoked.
- The environment itself needs no rollback — recreate it. That is the entire value proposition.
- Rolling back the *move* to ephemeral environments is harder: teams delete the long-lived environment once the new path works, and if the new path turns out to be flaky, the fallback is gone. Keep the old one until the new path has been exercised under pressure.
- Data seeded into an ephemeral environment is not recoverable after teardown, which surprises anyone who was using one as a scratch workspace. Say so loudly, in the environment itself.
- Automate creation, seeding, credential issuance, time-to-live enforcement, teardown and orphan sweeping. This is the module's clearest case of removing toil (Toil).
- Automate an orphan report even if deletion stays manual — knowing is most of the value.
- Do not automate deletion of anything holding data someone might need without a grace period and a notification. An automated reaper with no warning is an incident generator.
- The upfront cost is substantial and lands entirely on the platform side, while the benefit shows up as absence of cost elsewhere — a hard case to fund.
- Fast creation usually means cutting fidelity, so ephemeral environments trend toward the cheap end of the parity spectrum (Parity That Is Worth Paying For).
- Elastic environments make the bill spiky and harder to forecast than a known set of permanent ones.
- Nothing persists, so the informal practice of leaving useful test state lying around stops working, and some workflows genuinely depended on it.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe requirement — everything codified, including data and credential access — holds regardless of platform. What differs is how much is codifiable: a stack of managed cloud primitives is largely creatable by API, while a system with an on-premise appliance or a contractual third-party account has steps that no code can perform.
- CLOUD-SPECIFICTeardown completeness differs by provider and by resource. Some resources are deleted with their parent, others are deliberately retained to prevent data loss — disks, snapshots, static addresses and log buckets commonly survive. Providers also differ on which deletions are asynchronous, so a "successful" teardown can leave resources alive for some time. Verify with a tag-filtered inventory, not with the teardown command's exit code.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — treating environment creation itself as a tested capability with a success rate, not as a script.