The question this answers
Where does a credential live when the workload cannot get one from an identity provider?
The application needs a database password and a payment-provider API key. Neither can be minted from a workload identity. Both must be readable by exactly one service, changeable without a redeploy, and revocable in minutes when an engineer leaves or a laptop is stolen.
A single audited store with per-secret access policy, versioned values, programmatic rotation, and a read path that leaves a trail — replacing environment variables, configuration files and the one Slack message from 2022.
What a secret manager is actually for
The Security Engineering domain teaches secret management as a discipline: classification, lifecycle, rotation strategy, revocation, the whole Secrets Management treatment. This lesson is the infrastructure view of the same thing, and it answers a narrower question — what component holds the value at runtime, how does the workload get it, and what does that path cost.
A secret manager is not just encrypted storage. Encrypted storage solves one of five problems. The other four are what make it worth a dependency: access is per-secret policy rather than per-file permissions, so one service reads one secret; rotation can change the value without a deploy, because the workload fetches rather than embeds; audit records every read, which is the only way to answer "who had this" during an incident; and revocation is a policy change that takes effect on the next fetch, rather than an archaeology project across every place the value was copied.
Contrast that with the alternatives it replaces. An environment variable is visible in the process listing, in a crash dump, in a debug endpoint that prints the environment, and in the deployment manifest that was committed. A configuration file is a file, with file permissions, on every replica. A repository secret is text in a system whose history is very hard to erase. None of these can answer "who read this last Tuesday", and none can be revoked without a deployment.
What a secret manager does *not* solve: the application still ends up with the plaintext value in memory, and anything that can read that process can read the secret. It moves the boundary; it does not remove it. That honesty matters, because teams sometimes treat adoption as the end of the problem rather than as the start of a lifecycle.
The five questions, and the one everybody fails
Every secret in the system should have an answer to five questions. Four of them are usually fine after adopting a manager. The fifth — rotation — is where implementations quietly stop.
Rotation fails not because changing a value is hard but because of the overlap problem. At the instant you change a database password, some replicas hold the old one and some hold the new one, and both must work or you have caused an outage. The correct pattern is two active credentials: create the new one, wait for every consumer to pick it up, verify nobody is using the old one — from the audit trail, not from belief — and only then delete it. Systems that support two active credentials rotate reliably; systems that support one turn every rotation into a coordinated deploy, which is why those rotations get scheduled for next quarter, permanently.
The second reason rotation fails is caching. A workload that fetches the secret once at startup will keep using the old value until it restarts, which means rotation silently depends on a deploy. A workload that fetches on every use adds latency and load to the secret manager. The usual answer is a short in-process cache with a refresh, plus explicit handling of an authentication failure as a trigger to refetch rather than as a fatal error.
| Question | Environment variable | Encrypted config in the repo | Secret manager |
|---|---|---|---|
| Where is it stored? | In the manifest, and in every process listing | In version control, encrypted | In a managed store, encrypted under a key you control |
| Who may read it? | Anyone who can read the deployment or the process | Anyone who can read the repo and holds the decryption key | Exactly the identities the secret policy names |
| How does it rotate? | A redeploy | A commit and a redeploy | A programmatic change with two active versions, no deploy |
| Who read it last? | Unknowable | Unknowable | In the audit trail, with identity, time and source |
| How fast can it be revoked? | As fast as you can redeploy every consumer | As fast as you can rotate and redeploy | Next fetch, after one policy change |
A secret's life, and where it leaks
Walking the lifecycle end to end shows that most real leaks happen at the edges — not in the store, but in what the application does with the value after it has it, and in what nobody does at the end.
The store is rarely the weak point. The weak points are: a value logged during a debugging session; a crash dump or error report containing the process environment; a secret fetched by a workload whose role is broader than the one secret it needs; and the final state, where a secret that is no longer used is also never deleted, so it sits in the store for years, still valid, still granting access, appearing in no dependency graph.
That last one deserves its own alert. A secret nobody has read in ninety days is either dead — in which case delete it, because a valid unused credential is pure risk — or it is used by something on a quarterly schedule that will break when you delete it. Both answers are worth knowing, and neither is discoverable without the audit trail.
- 1Create
The value is generated — ideally by the system that owns it, never by a human choosing it — and written to the store with an owner and a purpose recorded.
A human-chosen value, or a value that passes through a chat message on its way into the store.
- 2Grant
A per-secret policy names exactly which identities may read it. One service, one secret.
A role granted
secrets:GetSecretValueon*— every secret in the account, from one workload. - 3Fetch
The workload authenticates with its own identity and reads the current version at startup, with a short cache and a refresh.
Fetch-once-and-cache-forever makes rotation silently depend on a redeploy.
- 4Use
The plaintext lives in process memory and is used to authenticate to the database or the third-party API.
Logged in a debug statement, included in an error report, or exposed by an endpoint that prints configuration. This is where most leaks happen.
- 5Rotate
A new version is created; both versions are valid; consumers pick up the new one; the audit trail confirms the old one is unused; the old version is disabled.
Single-credential rotation with no overlap, which turns every rotation into a coordinated outage risk — so it never happens.
- 6Revoke
On compromise: invalidate at the source system, remove read access, and read the audit trail to bound who fetched it and when.
No audit trail, so the blast radius is "everything that could have read it" rather than a list.
- 7Retire
When the consumer is gone, the secret is invalidated at the source and deleted from the store.
Never happens. Valid credentials for decommissioned systems accumulate indefinitely.
Key points
- Use a secret manager for the credentials that genuinely cannot be workload roles — start by eliminating the ones that can. See Roles vs Static Keys.
- It solves five things: storage, per-secret access, rotation without a deploy, an audit trail of reads, and fast revocation.
- Rotation requires two simultaneously-valid credentials. Without overlap, every rotation is an outage risk and therefore never happens.
- Fetch at startup with a short cache and treat an authentication failure as a signal to refetch, not as a fatal error.
- Most leaks are not in the store — they are in logs, crash dumps and error reports containing the plaintext.
- A secret nobody has read in ninety days is either dead or on a quarterly job. Find out which, then delete or document it.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The secret is stored encrypted under a key from a key management service, with each write creating a new version.
- • A per-secret policy names which identities may read it; the workload authenticates with its own short-lived role credential.
- • The workload fetches the current version at startup, caches it briefly, and refreshes on a timer or on an authentication failure.
- • Every read is written to the audit trail with identity, timestamp and source address.
- • Rotation creates a new version while the previous one stays valid, and the old version is disabled only after the audit trail shows nobody is using it.
- • Own the per-secret access policy.
secrets:GetSecretValueon*defeats the entire point of adopting a manager. - • Own rotation actually running, with evidence. An unrotated secret in a manager is a well-organized static key.
- • Own the refetch path in application code — this is where rotation succeeds or silently fails.
- • Own log hygiene: redaction in the logging layer, and a scanner that fails a build when a secret-shaped string appears.
- • Own retirement, driven by a "not read in 90 days" report, so dead credentials do not accumulate.
- • Rotation without overlap: half the fleet holds the old credential, and every request from those replicas fails authentication.
- • A workload that fetched once at startup and never refreshes, so it keeps using a revoked credential until it restarts.
- • A secret printed into a log line during debugging, which now lives in the log store with a much longer retention and a much broader access policy than the secret manager.
- • One role with read access to every secret, so a single compromised workload yields the entire credential set.
- • The secret manager becoming a hard startup dependency: it has a brief outage and nothing in the platform can start, including the things needed to fix it.
- • A secret that outlives its consumer, valid and forgotten, still granting access years later.
- • Secret count grows with services × environments, and per-secret policy is what keeps that growth safe rather than merely tidy.
- • Fetch volume is usually trivial — startup plus refresh — unless someone fetches per request, which turns the manager into a latency dependency.
- • The manager becomes a critical path component: rate limits and availability now matter to your startup path, which argues for caching and graceful degradation.
- • What runs out first is ownership. A store with 400 secrets and no owner per secret cannot be rotated or retired by anyone.
- • The plaintext still reaches process memory. A secret manager moves the boundary from "on disk everywhere" to "in memory in one process", which is a large improvement and not an elimination.
- • Scope read access to one secret per identity; broad secret access is the single fastest privilege escalation in a cloud account.
- • Encrypt under a key you control so that key deletion and key access are separate, auditable controls — see Key Management and Encryption at Rest.
- • The audit trail is the only mechanism that bounds a leak. Without it, "which secrets were exposed" has no answer smaller than "all of them".
- • Prefer per-environment secrets. A staging credential that also works in production erases the boundary between the two.
- • For agent and model workloads, treat provider credentials as first-class secrets and never let untrusted input reach the code path that reads them.
- • Priced per secret stored and per API call at most providers — trivial for hundreds of secrets, noticeable if something fetches per request.
- • The real cost is engineering: wiring fetch and refresh, implementing rotation with overlap, and building the log-redaction habit.
- • It avoids the cost of a credential-leak incident, where the expensive part is not rotating the secret but determining what was exposed.
- • Running your own secret store is a real alternative and moves the cost from a bill to an operational burden — including its own availability and backups.
- • Secret reads by identity — an identity reading a secret it has no business with is a high-signal alert.
- • Age of each secret against its rotation policy, which is the report that reveals rotation was never actually implemented.
- • Authentication failures against the database or third-party API immediately after a rotation, which is the fingerprint of a missing overlap.
- • Secrets not read in 90 days, for retirement.
- • The signal that lies: the secret manager's own health metrics. They are green while a workload happily uses a credential that was revoked last week.
- • Eliminate the secret entirely with a workload identity. Always the first move — the best-managed secret is one that does not exist. See Roles vs Static Keys.
- • The orchestrator's native secret object for a small cluster, understanding that it is encrypted at rest but has coarse access control and no rotation or audit. See ConfigMap vs Secret — and the Honest Limit of a Secret.
- • A managed database's identity integration, which replaces the password with a short-lived token and removes the highest-value secret from the store.
- • For a single-developer prototype, an environment variable from an untracked file is a reasonable, explicitly temporary answer — as long as it is not the production answer.
- • A self-hosted secret store when policy requires it, accepting that you now own its availability, its backups and its own root credential.
- • Buys per-secret access, audited reads and deploy-free rotation; costs a runtime dependency in your startup path.
- • Buys fast revocation; costs application work — refresh logic and treating authentication failure as recoverable.
- • Buys a single inventory of credentials; costs an ownership discipline that has to be maintained per secret, forever.
- • Buys a much smaller exposure surface; does not remove the plaintext from process memory, logs or crash dumps, which remain your responsibility.
The lifecycle of one database credential
| If it leaks | Exposure window | How you find out |
|---|---|---|
| static credential, no expiry full database access from anywhere on the internet that can reach the endpoint | until someone notices and dares to revoke it — typically months | a scanner, a bill, or a customer |
| short-lived token, 1 hour the same access, for one hour, from one identity that the log names | at most 60 minutes, then it is refused with no action from you | the issuing log shows the unexpected request |
What people believe, and what is true
We moved the secrets into a manager, so secrets are handled.
Storage was one of five problems. Without rotation, per-secret access and retirement, you have a well-organized collection of permanent credentials.
The application should fetch the secret on every use so rotation is instant.
That makes the manager a per-request dependency and a latency source. Cache briefly and refetch on authentication failure.
Encrypted secrets in the repository are equivalent.
Every holder of the decryption key can read every secret, rotation requires a commit and a deploy, and nothing records who read what.