Secrets Are Not Configuration
Credentials need a different storage, a different access path and a lifecycle — and rotation is the part every team skips.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Where do database passwords, API keys and signing keys actually live, and how do they change?
The service needs a database password, a payment provider key and a JWT signing key. None of them may end up in the repository, and when one leaks we must be able to replace it in minutes rather than days.
Put them in environment variables set by the deployment platform, and keep a .env file locally that is in .gitignore. It is simple, it works, and nothing is committed.
.gitignore protects the next commit, not the last one. Secrets get committed before the ignore rule exists, in a hurry, or by someone using git add -f — and git history is permanent and cloned by everyone.
.gitignoreprotects the next commit, not the last one. Secrets get committed before the ignore rule exists, in a hurry, or by someone usinggit add -f— and git history is permanent and cloned by everyone.- The platform's environment variables are visible to anyone with deploy access, printed in build logs, inherited by child processes, and included in crash dumps sent to error trackers (Secrets in Logs).
- There is no rotation story. Nobody knows every place the value was pasted, so changing it means an outage of unknown scope — which is why it never happens.
- There is no audit trail. When a key is suspected leaked, nobody can say who read it, when, or from where.
- The same value is shared by every environment and every engineer, so a laptop compromise is a production compromise (Least Privilege).
What is actually happening
- A secret differs from configuration in four properties, and each one implies different machinery: it grants access, it must be auditable, it must be revocable, and it must be rotatable without downtime.
- A secret manager is a service that stores secrets encrypted, authenticates callers as *workloads* rather than as people, authorizes each read by policy, logs every access, and supports versioning so a rotation can be staged (Secrets Management).
- The bootstrap problem is the interesting part: to read a secret you need a credential, so what authenticates the process? The answer is workload identity — the platform vouches for the running instance, and the secret manager trusts the platform rather than a stored key (Human vs Workload Identity).
- Rotation has two phases and the second is what gets skipped: create the new credential and make it work alongside the old, then verify nothing uses the old one and revoke it. Skipping revocation means the old credential still grants access, which is the state most systems are actually in.
- Overlap is what makes rotation non-disruptive. Two valid database users, two valid API keys, or two valid signing keys where verification accepts both and signing uses only the new one (Token Authentication and the Revocation Problem).
- Short-lived credentials are the structural fix: a credential that expires in an hour cannot be leaked for long, and rotation stops being an event because it is continuous (Short-Lived Credentials).
- Blast radius is set by scoping. One credential per service per environment with the narrowest permissions means a leak is bounded; one shared admin credential means a leak is total (Database Privileges and Blast Radius).
The lifecycle, not the storage
Teams solve "where do we put it" and consider secrets done. The stages that follow are where the risk actually lives, and each has a specific failure that is common enough to be predictable.
Note that revocation is a separate stage from rotation. Collapsing them is the single most common gap: a new credential is issued and adopted, the ticket is closed, and the old credential remains valid for years.
- 1Create
Generate with the narrowest scope that works — one service, one environment, minimum grants.
fails by Reusing an existing admin credential because it is already there and definitely works.
- 2Store
Write to a secret manager, encrypted, with an access policy naming the workload identity.
fails by A .env file, a wiki page, a chat message, or a CI variable with no policy.
- 3Distribute
The workload authenticates with a platform identity and fetches at boot into memory.
fails by A static bootstrap key used to fetch secrets — the same problem, one level down.
- 4Use
Held in memory only; never logged, never in a URL, never in an error, never on disk.
fails by A crash dump or error-tracker payload carrying the whole environment (Not Leaking Your Internals).
- 5Rotate
Issue a new version; both old and new valid; roll instances onto the new one.
fails by A rotation that requires simultaneous restart, converting a security task into an outage.
- 6Verify
Watch usage of the old version fall to zero across every consumer.
fails by Assuming zero because the deploy finished — a forgotten cron job still uses it.
- 7Revoke
Disable the old version. Only now is the rotation real.
fails by Never happening. This is the stage that is skipped, and skipping it means nothing was rotated.
- 8Audit
Every read logged with identity, version and time; anomalies alerted.
fails by Audit enabled but never queried, so the trail exists and nobody reads it (Audit Logs for Privileged Actions).
Rotation without downtime needs overlap in the code
Non-disruptive rotation is a code property before it is an operational procedure. If a client can only hold one credential, then there is an instant when some instances have the new one and some have the old, and one of those groups is broken.
The general shape: accept both during a window, prefer the new one for new work, and expose a metric that shows the old one falling out of use. Verification then has an answer rather than an assumption.
1// Two keys held at once. 'current' signs; both verify.2type KeySet = { current: { kid: string; key: Uint8Array }; previous?: { kid: string; key: Uint8Array } }3 4let keys: KeySet = await loadKeysFromSecretManager()5 6// Re-fetch periodically so a rotation propagates without a deploy.7setInterval(async () => { keys = await loadKeysFromSecretManager() }, 60_000).unref()8 9export function sign(claims: Claims): string {10 // Always the newest key, and stamp which one, so verify can pick.11 return jwtSign(claims, keys.current.key, { keyid: keys.current.kid })12}13 14export function verify(token: string): Claims {15 const kid = readKid(token)16 const match = [keys.current, keys.previous].find((k) => k?.kid === kid)17 if (!match) throw new AppError('authentication', 'unknown_key', 'Token key not recognised.')18 19 // The metric that makes revocation safe: when this hits zero for the old20 // kid across every instance, the old key is genuinely unused.21 tokenVerifications.inc({ kid })22 return jwtVerify(token, match.key)23}The previous slot must stay populated for at least the maximum lifetime of an issued token. Revoking earlier logs out every user holding a token signed by the old key — a rotation that becomes a visible incident.
Where secrets actually leak
Almost no secret leaks from the secret manager. They leak from the places the value travels afterwards, most of which nobody classified as an output channel.
Each row below has caused real disclosures. The pattern is consistent: a system designed to capture context faithfully captures a credential, and the capture is a feature.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Unhandled exception with the environment attached | Database password in a third-party error tracker | Error SDK captures the process environment by default | Configure an allowlist of captured context before the SDK reaches production (Secrets in Logs) |
| Debug logging switched on during an incident | Authorization headers in the log index | Debug level logs whole request objects | Redact at the logger, and never make full-object logging reachable by a flag (Structured Logging) |
| A credential passed in a query string | Key in access logs, proxy logs and browser history | URLs are logged everywhere by design | Credentials go in headers or bodies, never in URLs |
| A subprocess spawned for image conversion | Full parent environment inherited by a third-party binary | Environment inheritance is the default | Pass an explicit minimal environment when spawning (Command Injection) |
| CI pipeline echoing a variable for debugging | Secret in a build log readable by everyone with repo access | Masking only covers exactly-matching strings, not derived or encoded ones | Never echo; scope CI secrets per job and audit build-log access (CI/CD Security) |
| A production dump copied to a laptop | Live credentials on unmanaged machines | No sanctioned low-friction path to debug with real data | Provide brokered, short-lived, audited access (Short-Lived Credentials) |
How to build it
Most important first.
- Store secrets in a dedicated secret manager, never in the repository, never in a wiki, never in a chat message, and never in a plain platform environment variable if the platform offers a secret type (Secrets in Infrastructure).
- Authenticate the workload with a platform-provided identity rather than a stored bootstrap credential. Static keys used to fetch secrets recreate the original problem one level down (Roles vs Static Keys).
- Fetch at startup into memory, and re-fetch on a schedule or on an explicit signal if you support rotation without restart. Never write a fetched secret to disk or into a log.
- Scope narrowly: per service, per environment, with the minimum grants. A read-only reporting service should not hold a credential that can write.
- Design every secret for overlap from the start. If the code can only hold one API key, the rotation procedure requires downtime and will therefore not happen.
- Automate rotation on a schedule and *test it*. A rotation procedure that is only exercised during an incident is a procedure that fails during an incident.
- Scan the repository and its history in CI for secret-shaped strings, and add a pre-commit hook. Detection is the backstop for the discipline.
- Have a written revocation path per secret type: what to revoke, what breaks, how long it takes. Write it before you need it (Incident Response Lifecycle).
What can go wrong
- Rotation that adds the new secret and never removes the old, so the leaked credential remains valid indefinitely and everyone believes it was rotated.
- A rotation that restarts every instance simultaneously, causing an availability incident caused entirely by a security improvement.
- Secrets fetched at startup and cached forever, so rotating in the manager has no effect until the next deploy — and nobody knows which instances hold which version.
- The secret manager becoming a hard startup dependency with no retry, so its brief outage prevents the whole fleet from starting (Validate at Startup, Fail Loudly).
- The mitigation failing: a secret scanner tuned to known prefixes, missing a base64 blob or an internal token format that matches nothing it knows.
- A secret leaked through a path nobody classified as output — a stack trace, an error tracker payload, a support bundle, a debug endpoint, a core dump (Not Leaking Your Internals).
- Developers copying a production secret to a laptop to debug, which quietly makes every laptop a production credential store.
- A rotation races with in-flight work: a request that fetched the old credential and has not yet used it will fail if the old one is revoked at that instant. Overlap windows exist precisely to cover this (Atomic Operations).
- Concurrent refreshes of an expiring short-lived credential can stampede the issuer. Coalesce them so one refresh serves all waiters (Request Coalescing).
- Assume every secret will eventually leak. The design goal is to bound the damage — narrow scope, short life, fast revocation — not to guarantee it never happens (Defence in Depth).
- A committed secret must be treated as compromised the moment it is discovered, even in a private repository. Rewriting history does not undo the clones, the CI caches or the forks.
- Never log a secret, never include one in an error message, and never put one in a URL — URLs land in access logs, browser history, referrer headers and proxy logs.
- Separate humans from workloads. A person should not hold a production database password at all; they should get short-lived, audited access through a broker (Human vs Machine Identities).
- Signing keys are a distinct case: rotating one requires accepting both old and new during verification for at least the lifetime of the longest-lived issued token, or you invalidate every session at once (JWT — What It Is and What It Costs).
- Audit reads. An unusual access pattern against the secret manager is one of the highest-signal detections available, because legitimate workloads read predictably (Detection Engineering).
- "It is in an environment variable, so it is safe." Env vars are a delivery channel, not a protection. They are visible in process listings, inherited by children and captured by error trackers.
- "It is a private repository." Private repositories get cloned, forked, backed up, indexed by tooling and eventually made public by accident.
- "We rotated the key." Adding a new one is half a rotation. Until the old one is revoked and verified dead, nothing has been rotated.
- "Encrypted at rest means secure." It means the disk is protected. Anything with read access to the manager still reads plaintext, which is why access policy and audit are the real controls (Encryption at Rest vs in Transit).
- "We will rotate if we detect a leak." Detection is unreliable and slow. Assume the leak, bound the damage, and rotate on a schedule regardless.
- "Secrets in a private config repo, encrypted with a key in another repo, is fine." It moves the problem to the key and usually stops there.
Operating it
- Alert on secret-manager access from an unexpected identity, an unexpected region or an unexpected rate. Legitimate access is boringly regular.
- Track the age of every secret. An age graph makes the rotation conversation concrete and shows the ones nobody has touched since the service was created (The Secret Lifecycle).
- After a rotation, watch usage of the *old* credential and only revoke when it reaches zero. That metric is the entire safety mechanism for a non-disruptive rotation.
- Log secret *fetches* — identity, secret name, version, timestamp — never values. This is the audit trail that makes an incident investigable.
- Run history scanning in CI and alert on findings. A hit is an incident, not a lint warning.
- At 10x instances, a startup fetch becomes a thundering herd against the secret manager during a rolling deploy. Cache, jitter the fetch, and treat the manager as a dependency that needs a retry policy (Backoff and Jitter).
- At many services, per-service scoping stops being optional: a shared credential means any one service's compromise is every service's compromise (Failure Propagation).
- Manual rotation does not survive growth. Somewhere between five and fifty secrets it stops happening at all, which is the argument for automation and for short-lived credentials.
- Multi-region adds a replication question: where is the secret stored, what is the latency to read it, and what happens to a region that is cut off from the manager (Multi-Region Deployment).
- A secret manager is another runtime dependency in the startup path. It must be treated as one, with timeouts, retries and a clear failure behaviour.
- Short-lived credentials remove most of the leak risk and add refresh logic, clock sensitivity and a new class of "the credential expired mid-operation" failure.
- Overlap-capable rotation means the code must hold and try more than one credential, which is real complexity in every client that uses one.
- Strict separation of human and workload access makes debugging genuinely slower. That friction is most of the security benefit, and it needs a sanctioned fast path or people will route around it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe four properties — grants access, auditable, revocable, rotatable — hold regardless of stack or platform.
- CLOUD-SPECIFICManaged secret stores differ in ways that change your design, not just your API calls: some support automatic rotation with a provider-supplied rotation function for managed databases, some only version and store; some inject secrets as files that update in place, others only as env vars fixed at start. Workload identity mechanisms are entirely provider-specific. Treat "our provider has a secret manager" as the beginning of the design, not the end (Secrets in Infrastructure).
- SCALE-SPECIFICFor a single service run by one team, a platform secret store with manual rotation is a defensible position. Above roughly a handful of services and engineers, manual rotation stops happening and short-lived, automatically-issued credentials become the only approach that survives.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.