The question this answers
If everything is encrypted at rest by default, what is actually left for me to decide?
Customer records must be encrypted at rest, an auditor must be shown who can decrypt them, and a departing customer's data must be provably unrecoverable. "The provider encrypts it" answers none of these.
A key hierarchy with an access policy of its own: encryption that costs the application nothing, plus a separate, auditable control over who may decrypt and an operation that renders data unrecoverable without touching the data.
Envelope encryption, and why there are two keys
Encrypting a 40 GB volume directly with a key held in a hardware security module would mean streaming 40 GB through that module. Nobody does this. Instead a data encryption key is generated locally, used to encrypt the data at full speed, and then itself encrypted by a key encryption key that never leaves the key management service. The encrypted data key is stored next to the ciphertext. To read the data you send the encrypted data key to the service, and it returns the plaintext data key — if, and only if, your identity is permitted to use that key.
That indirection is the whole design, and it buys three things. Bulk encryption stays fast, because it happens locally with a symmetric key. The high-value key never exists outside a controlled boundary. And access is a policy decision, evaluated per request — which means "who can read this data" becomes an IAM question with an audit trail, rather than a question about who has a copy of a key file.
The cryptographic primitives underneath — the ciphers, the modes, the key derivation, why symmetric encryption is used for bulk data — belong in Security Engineering (Symmetric Encryption, Cryptography Fundamentals). This lesson deliberately stops at the boundary, because in infrastructure practice you will never choose a cipher and you will absolutely choose a key policy.
Three operational questions, and their real answers
Who can decrypt? Not "who has the key" — nobody has the key. The answer is: every identity whose policy grants Decrypt on that key. This is the question auditors ask and the one teams answer badly, because a key created with a default policy is frequently usable by every identity in the account, which makes encryption at rest a compliance checkbox rather than a control. Encryption at rest defends against a stolen disk and against a provider employee; it defends against a compromised workload only to the extent that the key policy denies that workload.
What does key deletion mean? It means the data is gone. Deleting a key encryption key renders every data key under it permanently unwrappable, and therefore every ciphertext permanently unreadable — including backups and snapshots. This is genuinely useful: crypto-shredding is how you make a departed customer's data unrecoverable without locating every copy of it. It is also irreversible, which is why providers impose a mandatory waiting period before deletion actually happens, and why a key deletion should require more approval than almost any other infrastructure operation.
What does rotation actually re-encrypt? Usually nothing. Rotating a key encryption key creates new key material and uses it for *future* wraps; existing data keys stay wrapped under the previous version, which remains available for decryption. Your 40 TB of existing objects are not rewritten, and nothing about them changes. That is a sensible design and it is not what most people picture. If the requirement is that old data becomes unreadable under the old key material, that requires re-encrypting the data — a separate, expensive, application-level project. Being clear about this distinction is the difference between a rotation policy that satisfies an auditor and one that satisfies a threat model.
| Control | Protects against | Does not protect against | What you operate |
|---|---|---|---|
| Provider-managed key | Stolen physical media; provider-side access | A compromised workload that is allowed to decrypt | Almost nothing — it is on by default |
| Customer-managed key | The above, plus any identity not named in the key policy | An attacker who compromises an identity that *is* named | The key policy, rotation and deletion — the real work |
| Key policy narrowed to one role | Lateral movement: another compromised service cannot decrypt this data | The service that legitimately holds the permission | Per-key policy review, alongside IAM review |
| Key rotation | Long-term exposure of one key version's material | Anything about already-written data, which stays under the old version | A schedule, and honesty about what it does not do |
| Key deletion (crypto-shredding) | Recovery of data from any copy, anywhere, including backups | Nothing — it is total, which is the risk | A heavily-approved, waiting-period-protected procedure |
| Encryption in transit | Interception on the network path | Anything at rest, and anything after termination at the load balancer | Certificates and their expiry — see Load Balancers as Infrastructure |
One read, hop by hop
Following a single decrypt makes the failure modes obvious. The service asks storage for an object; storage holds the ciphertext and the wrapped data key; the key service is asked to unwrap it *using the caller's identity*; the unwrapped data key decrypts the object in memory and is discarded. Every unwrap is an authorization decision and every one is logged.
Two consequences follow immediately. First, the key service is on the read path — a key-service throttle or outage presents as your data being unreadable, which looks like a storage failure and is not one. Caching the unwrapped data key for a short window is how systems avoid an unwrap per object, and it is also how a permission revocation takes effect later than people expect.
Second, cross-region and cross-account boundaries become key boundaries. A key is regional; a snapshot encrypted in one region cannot be restored in another without a key that exists there and a policy that allows it. This is the specific, unglamorous reason a cross-region disaster-recovery plan fails its first real test — the data replicated correctly and nothing in the second region is permitted to decrypt it. Check it as part of Restore Testing, not as part of the incident.
Key points
- Envelope encryption: a fast local data key encrypts the data, and a key that never leaves the key service encrypts the data key.
- What you operate is the key policy, not the cryptography. "Who can decrypt" is an IAM question with an audit trail.
- A key with a default, account-wide policy makes encryption at rest a checkbox rather than a control.
- Deleting a key destroys every ciphertext under it — including backups. That is crypto-shredding, and it is irreversible by design.
- Rotation creates new key material for future wraps; it does not re-encrypt existing data, and pretending otherwise misleads an audit.
- Keys are regional and account-scoped, which is why cross-region restores fail on decryption rather than on copying.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A data encryption key is generated for the volume, object or database and used locally to encrypt the bytes.
- • The data key is sent to the key service, wrapped under a key encryption key that never leaves the service boundary, and stored alongside the ciphertext.
- • On read, the wrapped data key is presented to the key service; the caller's identity is evaluated against the key policy.
- • If permitted, the plaintext data key is returned, held briefly in memory, used to decrypt, and discarded.
- • Every wrap and unwrap is recorded in the audit trail with identity, key, time and source — see Audit Trails.
- • Own the key policy per key, and review it exactly as seriously as an IAM policy. This is the entire control surface.
- • Own the decision between provider-managed and customer-managed keys per data class, and be able to state why.
- • Own key deletion as a heavily-approved procedure with a waiting period. It is the most irreversible operation in your account.
- • Own the cross-region key story before a disaster-recovery test, not during one.
- • Own the rotation schedule and, more importantly, an honest statement of what it does and does not change.
- • A cross-region restore that fails because no key exists in the target region, or its policy does not permit the restoring identity — usually discovered during the first real recovery.
- • A key policy so broad that every identity in the account can decrypt, so encryption at rest stops defending against anything you actually face.
- • A key disabled or scheduled for deletion, taking every service that depends on it offline simultaneously.
- • Key-service throttling under a burst of unwraps, presenting as widespread read failures that look like a storage outage.
- • Permission revoked but data still readable for the cache window, because unwrapped data keys were cached.
- • A snapshot shared to another account without the key, producing a copy that is useless — or, worse, shared *with* key access, producing a copy that is not.
- • Bulk encryption throughput is not a constraint; it happens locally at symmetric-cipher speed.
- • Key-service request rate is the real constraint, and envelope encryption plus a short data-key cache is what keeps it off the hot path.
- • Key count grows with data classes and environments, and each key is a policy someone must review — this is the dimension that runs out.
- • Cross-region replication multiplies keys and policies, which is where the operational cost of encryption actually lives.
- • Encryption at rest defends against stolen media and provider-side access. It defends against a compromised workload only insofar as the key policy excludes that workload.
- • Separate key administration from key use: the identity that may
Decryptshould not be the identity that may change the key policy. - • Use different keys for different data classes so that a policy mistake on one does not expose all of them.
- • Key-use logs are among the highest-signal events available — an identity decrypting data it has never decrypted before is worth an alert.
- • Crypto-shredding is a legitimate and powerful deletion mechanism for data-subject requests; treat the key as the thing that must be inventoried, because it is now the thing that makes deletion real.
- • Encryption in transit is a separate control on a separate path; neither substitutes for the other. See Encryption at Rest vs in Transit.
- • Key storage is a small per-key monthly charge; key operations are priced per request and become visible only at high unwrap rates.
- • Envelope encryption plus caching is what keeps the request meter irrelevant — a per-object unwrap at scale is where surprises come from.
- • Customer-managed keys cost slightly more than provider-managed ones and buy the policy control that is the entire point.
- • The largest real cost is human: reviewing key policies and running a cross-region key story, both recurring.
- • Decrypt calls by identity and by key — a new identity decrypting a sensitive key is a genuine security signal.
- • Key-service error and throttle rates, which manifest upstream as data being unreadable rather than as a key problem.
- • Pending key deletions, alerted loudly, because the waiting period is the only chance to stop an irreversible mistake.
- • Key policy changes as first-class events, alongside IAM policy changes.
- • The signal that lies: "encryption at rest: enabled". It is enabled in the account where every identity can decrypt, and in the one where exactly one role can.
- • Provider-managed default encryption. Free, on by default, no key policy to maintain, and adequate for most data classes — the honest right answer far more often than a customer-managed key.
- • Application-level encryption of specific fields when the threat model includes your own database administrators, accepting that you lose indexing and querying on those fields.
- • No encryption beyond the default for genuinely public data. Encrypting a public website's static assets under a customer-managed key adds a dependency and protects nothing.
- • Deleting the data. If the requirement is that data becomes unrecoverable, actual deletion is simpler and more provable than key management for most systems.
- • External key management, when regulation demands you hold the material outside the provider — at the cost of a hard availability dependency you now own.
- • Customer-managed keys buy an auditable, narrow answer to "who can decrypt"; they cost a policy to maintain and a new hard dependency on the read path.
- • Crypto-shredding buys provable, total deletion; it costs irreversibility, and a mistaken key deletion is unrecoverable data loss.
- • Per-data-class keys buy isolation between policy mistakes; they cost more keys, more policies and more cross-region plumbing.
- • Caching unwrapped data keys buys throughput and cost control; it costs immediacy of revocation for the length of the cache window.
What people believe, and what is true
Encrypted at rest means our data is protected.
It is protected against stolen media and provider-side access. Against a compromised workload it protects exactly as much as the key policy excludes that workload.
Rotating the key re-encrypts our data.
It creates new material for future wraps. Existing ciphertext stays under the previous version. Re-encryption is a separate, expensive project.
We can restore this snapshot in another region.
Not without a key in that region and a policy that permits the restoring identity. This is a standard disaster-recovery failure, and it fails at decryption, not at copying.
Deleting a key is like deleting a file.
It permanently destroys every ciphertext under it, backups included. That is why it is useful, and why it needs a waiting period and real approval.