AccessORG-SPECIFICCLOUD-SPECIFICSCALE-SPECIFIC
Production Access
Who can reach production, what they can do there, and why the answer is a set of specific approved capabilities rather than a single administrator role.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
What access does an engineer actually need to operate a production system, and why is "admin" almost never the right answer?
The problem
Operating production requires reaching it: reading logs, checking state, restarting a stuck consumer, rolling back a bad release. Granting that has historically meant granting everything, because the capability to look and the capability to destroy arrive in the same role.
What teams do first
Engineers need to be able to fix production, so give the team administrator access. Restricting it just means people cannot do their jobs during an incident.
How it breaks
A single mistyped command reaches everything. The blast radius of an operator error equals the blast radius of the role, and admin roles have no boundary (Blast Radius: If This Is Wrong, How Much Does It Affect?).
How it breaks in production
- A single mistyped command reaches everything. The blast radius of an operator error equals the blast radius of the role, and admin roles have no boundary (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Every credential becomes a total compromise. A phished session, a stolen laptop, a leaked token in a build log — each one is now equivalent to full production control (Secrets in CI).
- Audit records become useless. When everyone can do everything, the logs record that someone did something, and the interesting question — was this person supposed to be able to do that? — has no answer (The Audit Trail).
- Access accumulates and never leaves. People join projects, get granted access, move on, and keep it. After two years the set of people who can drop the production database is nobody's intent and everybody's reality (Access Review).
- Broad access makes manual production changes easy, which is how configuration drifts away from code without anyone deciding to do that (Manual Production Changes).
- When everything requires the same all-powerful role, it gets shared, left logged in, and embedded in scripts — because the friction of getting it is high enough that people work around it once.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The useful decomposition is not "access or no access". It is a set of capabilities with very different risk profiles: reading telemetry, reading application logs, reading data, executing a defined operation, changing configuration, changing infrastructure, changing data.
- Those capabilities are separable, and the everyday operational ones are overwhelmingly on the read-and-defined-operation end. Most production work is looking at things and running one of a small number of known operations.
- The right structure is therefore capability, not role: an operator can restart a deployment, scale it, roll it back, and read logs — and cannot delete a database, modify IAM, or read customer records. That set covers nearly all of on-call (Least Privilege in Production).
- Access has a time dimension as well as a scope dimension. Standing access is available to an attacker at all times; access granted on request for a bounded window is only available when it has been granted, and that shrinks the exposure window by orders of magnitude (Short-Lived Credentials).
- The residue — the genuinely rare case where someone needs more — is handled by an explicit, audited, expiring elevation rather than by making the everyday role bigger (Break-Glass Access).
- The other half of the answer is that most access requests are really tooling requests. "I need SSH to check the config" means the config is not visible anywhere else. Building the read path removes the access requirement entirely, which is the strongest form of least privilege (Dashboards an Operator Can Act On).
Capability, not role
GENERALThe decomposition holds on any platform. What differs is how cleanly it can be expressed: some IAM systems separate these naturally, others bundle infrastructure and data access into one role and force you to build the boundary in tooling instead.
The reason "admin" is the default answer is that the capabilities engineers actually need are bundled with the ones they do not, and unbundling them takes work. The unbundling is the whole lesson.
Read the last column. Almost everything an on-call engineer does in a typical month sits in the top four rows, and none of those rows can destroy anything.
| Capability | Typical operational need | Worst outcome if misused | Standing or on request? |
|---|---|---|---|
| Read telemetry | Every incident, constantly | Cardinality cost; occasionally sensitive labels | Standing |
| Read application logs | Most incidents | Exposure of anything logged carelessly (Secrets in Logs) | Standing, with redaction at write time |
| Execute defined operations | Restart, scale, roll back, toggle a flag | Bounded by the operation — a bad rollback is recoverable | Standing, scoped to owned services |
| Change configuration | Occasionally, under pressure | Wrong value applied everywhere at once (Configuration Drift) | Through the reviewed path; direct change on request |
| Change infrastructure | Rare outside planned work | Deleted resource with no state to recreate it (Destructive Changes: What a Rename Really Does) | On request, reviewed |
| Read production data | Rare, and less often than assumed | A breach with legal and contractual consequences | On request, time-boxed, always audited |
| Change production data | Very rare, always an emergency | Silent corruption that backups may already have captured | Break-glass only (Break-Glass Access) |
| Change access itself | Almost never during operations | Escalation to everything, including disabling the audit trail | Break-glass, with a second approver |
Three paths to the same operation
An engineer needs to roll back a bad release. There are three ways to give them that capability, and they differ enormously in what else they permit.
The third path is the one worth investing in. The capability is delivered by a system that can only do that operation, so no human ever holds a credential that could do anything else. It is also the only path that produces a usable audit record without extra work — the pipeline knows who asked, what for, and what it did.
- Path 1 — fastest to set up, no boundary at all. A typo reaches every resource in the cluster.
- Path 2 — bounded by verb and namespace. A typo reaches the services this team owns, which is a real reduction and not a small one.
- Path 3 — bounded by the operation itself. There is no typo that deletes anything, because deletion is not a capability the path has (Self-Service Infrastructure).
- Most organisations should be moving work from path 1 to path 3 over time, and keeping path 2 for the operations that are not yet worth building tooling for.
UserLLMAgentToolDataDecisionHumanGuardrail
How access models fail in both directions
The failures split evenly between too much and too little, and the too-little ones are less discussed because they surface as slow incidents rather than as breaches. Both are worth designing against.
Access failures, in both directions
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Everyone has admin | A single command affects everything | Capabilities never unbundled | Define an operator role from the operations on-call actually performs |
| Credential leaked in a build log | External access with full production rights | Long-lived credential with unbounded scope | Short-lived, scoped credentials; workload identity for automation (Short-Lived Credentials) |
| On-call cannot restart a service | Incident extended while someone finds an approver | Access restricted without mapping what on-call actually does | Grant the standing operator capabilities; restrict what is genuinely dangerous |
| Shared admin account | Audit log shows an action and no person | One credential used by many humans | Individual identity always; shared accounts make the trail unusable (Audit Logs for Privileged Actions) |
| Access granted for a migration project | Still active two years later | Grants have no expiry | Time-bound by default; expiry is the control, review is the backstop (Access Review) |
| Elevation approved in seconds, every time | A control that has never once refused | Approver has no context and no incentive to refuse | Route to someone with context; measure approval rate and time as a signal |
| Read access includes customer records | A leaked read-only session is a reportable breach | Telemetry read and data read bundled | Separate them; data access is time-boxed and audited even for reads (Sensitive Data Classification) |
| Automation runs as a human's credential | Jobs break when the person leaves; audit attributes machine actions to a person | No workload identity | Machine identity for machines, always (Human vs Workload Identity) |
How to do it properly
Most important first.
- Enumerate the operations on-call actually performs, then define a role that permits exactly those. In practice it is a short list: read telemetry, read logs, restart, scale, roll back, toggle a flag.
- Separate reading telemetry from reading data. Debugging almost always needs the first; it needs the second far less often than people assume, and customer data is the part with legal consequences (Production Data in Lower Environments).
- Make the safe path fast. If getting the right access takes twenty minutes and the admin credential is already in someone's terminal, the admin credential wins during an incident every time.
- Prefer capability through tooling over capability through credentials: a deploy pipeline that can roll back on request means nobody needs cluster write access to roll back (Self-Service Infrastructure).
- Use workload identity for automation rather than long-lived keys held by humans or CI (Workload Identity).
- Log every production action with actor, target, time and the reason it was permitted, into a store the actor cannot modify (Audit Logs for Privileged Actions).
- Review who has what on a schedule, and expire access that has not been used (Access Review).
- Keep exactly one documented path to more privilege, and make sure it is fast enough that nobody builds a second one (Break-Glass Access).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Contained by the scope of the role in use — which is exactly the point. An admin role contains nothing; a capability-scoped role contains an error to the resources it names.
What can go wrong
Failure modes, including of the mitigation
- Access so restricted that on-call cannot mitigate, so people acquire a shadow credential and the control exists on paper only.
- A permission model so complex that nobody can determine who can do what, which is functionally the same as having none (Anatomy of a Policy).
- Read access treated as harmless when it includes customer data — a data breach with no write involved at all.
- Service accounts with broad standing permissions used by humans, which defeats both the scoping and the audit trail.
- Elevation requests approved reflexively within seconds, which converts a control into a logging mechanism.
- Access granted for a project and never revoked, so the effective permission set is the union of everything anyone has ever needed.
- Break-glass used routinely because the everyday role is genuinely insufficient — at which point the everyday role is wrong and the emergency path has become the normal one.
Misreads this invites
- "Least privilege means engineers cannot fix production." It means they have the capabilities to fix production and not the capabilities to destroy it. Those are different sets, and the second is much larger.
- "Read-only access is safe." Read access to customer data is a breach waiting for a credential leak. Telemetry read and data read are different capabilities and should be granted differently (Sensitive Data Classification).
- "We are too small for this." A five-person team can restrict who holds the credential that can drop the database. Small teams have less recovery capacity, not more.
- "Access control is a security concern, not a delivery one." Access decides who can deploy, who can roll back and who can change config at 3am. It is squarely a delivery concern that security also cares about.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- You can answer "who can delete the production database?" from configuration, and the answer is a short list you recognise.
- On-call engineers complete a normal mitigation without elevating.
- Break-glass use is rare and each use has a recorded reason.
- Every production action appears in an audit log with an actor, including actions taken by automation.
- Access reviews remove permissions, rather than confirming them all.
How you get back
- Tightening access is reversible and should be staged: run the narrower role alongside the broad one, log what the broad one is still being used for, and remove it once the log is empty for a period. Cutting over blind produces an incident where nobody can mitigate.
- Keep the emergency path working throughout any access change. The failure mode of a permission migration is discovering during an incident that the new role is missing something, so the break-glass procedure is the safety net for the safety net.
What to automate, and what stays human
- Automate grant and expiry. Access that must be manually revoked is access that persists.
- Automate the audit trail so it does not depend on anyone recording anything.
- Automate the common operations into tooling — rollback, restart, scale, flag toggle — so the capability exists without the credential (Golden Paths).
- Do not automate approval of elevated access. A human deciding is most of the value, and an approval bot that always says yes is a log line with extra latency.
- Do not automate revocation mid-incident. An access grant that expires while someone is using it to mitigate an outage turns one incident into two (Break-Glass Access).
What this costs
- Every restriction adds friction to legitimate work, and the friction lands hardest during incidents, which is when errors are most costly and patience is lowest.
- Fine-grained permissions are more work to maintain and harder to reason about. There is a real point past which more granularity reduces safety because nobody understands the model.
- Building tooling so capability does not require credentials is the right answer and is a genuine engineering investment — often larger than the access work it replaces.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICAccess policy is shaped by company size, industry and regulator. A regulated financial platform may require separation of duties and named approvers by law; an early-stage startup with four engineers is choosing a convention. Both need someone to have decided; only one has the decision made for them.
- CLOUD-SPECIFICThe mechanism differs by provider — IAM policies, roles and resource scoping do not map one-to-one across AWS, GCP and Azure, and permission evaluation order differs. The capability decomposition transfers; the policy documents do not (§178).
- SCALE-SPECIFICBelow about ten engineers, a small trusted group with broad access and good audit logging is a defensible position. Above that, "everyone is trusted" stops being a control because you no longer know who everyone is.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Backendauthorization-in-backends
Domains that do not exist yet
- — System Design — the operational boundary between a service and the people who run it, as a design property rather than a policy layered on afterwards.