Reliability & Disaster Recovery

Backup Strategy

A backup is only a backup if it is in a separate failure domain. Same account, same region, deletable by the same credentials that just got compromised — that is a copy, and it does not survive the most common causes of data loss.

The question this answers

Infrastructure question

Where does the second copy live, who can delete it, and how far back can it take us?

Application requirement

Data must survive events that destroy the primary: an accidental deletion, a bad migration, a compromised administrator credential, ransomware, or the loss of the account itself. Replication does not survive any of those, because it copies them.

What it provides

An independent point-in-time copy in a different failure domain, protected from the credentials that control production, retained long enough to cover damage discovered late.

Application RequirementInfrastructure RequirementComputeNetworkStorageIdentityDeploymentScalingReliabilityObservabilitySecurityCostTrade-offs

Separate storage, separate failure domain

The defining property of a backup is independence. A snapshot in the same account, same region, deletable by the same role that manages production, is a convenience — excellent for recovering a table someone truncated, useless against the failure modes that actually destroy companies. The question to ask of any backup is not "does it exist" but "what disaster does it survive?"

Work down the list. A snapshot in the same region survives an accidental deletion of the source. A backup in a different region survives a regional loss. A backup in a *different account* with *different credentials* survives a compromised production administrator. A backup with immutable, write-once retention survives an attacker who obtained backup credentials too. Each step outward costs a little more and covers a strictly larger set of disasters, and most teams stop one step too early.

The direction of the copy matters as much as its location. If production has credentials that can write to and delete from the backup store, then whatever compromises production reaches the backups. The safer arrangement is a pull model, or a push into a vault whose retention policy cannot be shortened by the pushing identity. Ransomware operators look for backups first, precisely because most organizations have made them reachable from the environment being encrypted.

The separation that makes it a backup rather than a copy.PROVIDER-NEUTRAL
Production accountprivate
Primary databaseprivate
Object storage (user uploads)private— versioning + soft delete: the cheapest recovery there is
Local snapshots — same accountprivate
Production admin roleinternal
⚠ Full control of production. No path to the vault — that absence is the control.
Backup account — different regionprivate— separate credentials, separate blast radius
Immutable vault (write-once retention)private— cannot be deleted or shortened before its retention expires — including by you
Backup encryption keysprivate— must be available during recovery and must not be deletable by production
Primary databaseLocal snapshots — same account· scheduled snapshot
Primary databaseImmutable vault (write-once retention)· push, write-onlycrosses boundary
Object storage (user uploads)Immutable vault (write-once retention)· replication, versionedcrosses boundary
Production admin rolePrimary database· full control
Production admin roleLocal snapshots — same account· full control — including delete
Immutable vault (write-once retention)Backup encryption keys· encrypted with

Who can delete the backups?

This is the question that separates a backup strategy from a backup schedule, and it is an identity question rather than a storage one. If the answer is "the same role that runs production", then the blast radius of that role includes your recovery capability, and no amount of retention configuration changes it.

The policy panel below is the arrangement that holds up. The production workload can write backups and read nothing back. It cannot list, cannot delete, cannot shorten retention, cannot change the encryption key. Restoring is a deliberate act performed by a separate identity, ideally requiring a second person. Yes, that means a restore takes a few minutes longer to initiate; that is the correct trade against an attacker who has your production credentials.

The red flag to name explicitly: "it is a managed database, so we do not need backups." A managed database gives you automated snapshots inside your account, inside the provider's retention window, deletable by your administrators and lost with your account. It protects against hardware failure, which was never the risk. It does not protect against a bad migration discovered in three weeks, a compromised credential, or a billing suspension. Managed moves the boundary; it does not remove your half (Shared Responsibility).

The backup writer, scoped so a production compromise cannot destroy recovery.
backup-writer (attached to the production data tier)workloadleast privilege
on Backup vault in a separate account, immutable retention 35 days
Allowed
  • backup:PutObject (write new backup objects)
  • kms:GenerateDataKey (encrypt what it writes)
Actually needed
  • Write a backup. Nothing else.
Explicitly denied
  • backup:DeleteObject
  • backup:ListObjects
  • backup:PutRetentionPolicy
  • kms:ScheduleKeyDeletion
  • backup:GetObject

Blast radius: An attacker holding this identity can write junk backup objects and consume storage. They cannot read existing backups, cannot delete anything, and cannot shorten retention — so the recovery path survives a full compromise of production. Restore is performed by a separate identity in the backup account, which is where a second-person approval belongs.

Frequency, retention, encryption — and the one that is not optional

Frequency sets your RPO and nothing else: backups every hour means up to an hour of loss (RPO & RTO). Continuous log archiving turns that into point-in-time recovery, usually for far less than people expect, and it is the single best value in this lesson.

Retention is set by how long damage can go undetected, which is much longer than teams assume. A bad migration that corrupts a rarely-read column may surface in a quarterly report. Ransomware is often present in an environment for weeks before it acts. Seven days of retention is a plan that assumes you notice everything within a week. A layered schedule — hourly for a day, daily for a month, monthly for a year — costs surprisingly little once older tiers move to cold storage (Storage Lifecycle: Hot, Warm, Archive, Delete).

Encryption is mandatory and has a specific trap: the key must be available during a disaster and must not be destroyable by whatever destroyed production. A backup encrypted with a key that lived only in the account you lost is not recoverable, and this is a real, recurring incident. Manage backup keys in the backup account (Key Management and Encryption at Rest).

And the one that is not optional: restore testing. Every property in this section is a claim about a file. Only a restore turns it into a fact, and that is Restore Testing — the next lesson, and the one that decides whether any of this was real.

ParameterDecidesCommon mistakeReasonable default
FrequencyYour RPO — nothing elseNightly backups paired with a documented one-hour RPOContinuous log archive plus daily full backups
RetentionHow far back you can reach for damage found lateSeven days, which assumes you notice everything within a weekLayered: hourly/24h, daily/30d, monthly/12m
LocationWhich disasters it survivesSame account, same region — a copy, not a backupSeparate account, separate region
DeletabilityWhether a compromise takes the backups tooProduction admin can delete backupsImmutable retention; production identity has write-only access
Encryption + key locationWhether it is confidential *and* recoverableKey lives only in the account that was lostEncrypted, with keys managed in the backup account
CoverageWhat is actually backed upDatabase only — no object storage, secrets, DNS or IaC stateAn inventory reviewed whenever a stateful component is added
Restore testingWhether any of the above is trueNever doneScheduled drills with measured duration (Restore Testing)
The parameters, and what each one is actually deciding

Key points

  • A backup must be in a separate failure domain: separate account, separate region, separate credentials.
  • If the production identity can delete the backups, a production compromise destroys your recovery.
  • Write-only access plus immutable retention is what survives an attacker who has everything else.
  • "Managed database, so we do not need backups" is a red flag: provider snapshots live in your account, in your retention window, deletable by your admins.
  • Retention is set by how long damage can go unnoticed, which is far longer than a week.
  • The encryption key must be available during the disaster and not destroyable by it — keep it with the backup, not with production.
  • Coverage is always broader than the database: object storage, secrets, DNS, certificates, IaC state, third-party configuration.
  • None of it is real until a restore has been performed.

The loop, answered

Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.

How it works
  • A scheduled job or managed service produces a consistent point-in-time copy — snapshot, dump, or continuous log archive.
  • The copy is encrypted and pushed to a store in a separate account and region, using an identity with write-only permission.
  • Retention policy governs how long each copy lives; immutable or object-lock retention prevents early deletion by anyone.
  • Continuous log archiving between full backups enables point-in-time recovery to an arbitrary moment.
  • Lifecycle rules move older backups to colder, cheaper storage tiers as they age.
  • Restore reads the chosen copy back into a target environment, and only then does any of it count.
What you still own
  • Maintain the backup inventory as a living list. New stateful components arrive constantly and nothing automatically adds them.
  • Alert on backup age and on unexpected size changes — a backup that succeeded but halved in size is the classic silent failure.
  • Verify the separation periodically by asking the production admin role to delete a backup and confirming that it cannot.
  • Keep encryption keys and their access policies in the backup account, and test that recovery works with only backup-account credentials.
  • Review retention after every incident where damage was discovered late; that is the data point retention is supposed to encode.
  • Schedule the restore drills. Everything else here is preparation for a step nobody performs by accident (Restore Testing).
How it fails
  • Backups deleted along with production, because the same credential reached both.
  • Backup succeeded, contents unusable: a dump taken mid-transaction, a corrupted archive, a missing file that nobody validated.
  • Encryption key unavailable during recovery because it lived in the account that was lost.
  • Retention too short: the corruption began five weeks ago and the oldest backup is two weeks old.
  • Incomplete coverage: the database restored perfectly and the user uploads, secrets and DNS records were never backed up at all.
  • Restore never tested, so the first attempt happens during the disaster and fails for a reason nobody anticipated.
  • A silently broken schedule — the job errored for three months and the alert went to a mailbox nobody reads.
How it scales
  • Backup duration and cost grow with data volume; at some size full backups stop being feasible and incremental plus log archiving becomes mandatory.
  • Restore duration grows with volume too, and is what eventually breaks a documented RTO (RPO & RTO).
  • Retention cost grows with volume × retention period, which is why lifecycle tiering matters more as you grow.
  • Coverage complexity grows with the number of stateful components, and it is the dimension most likely to silently fall behind.
Security
  • Backups contain everything sensitive that production contains, with weaker access monitoring. They are a high-value target and are treated as an afterthought remarkably often (Backups Are Sensitive Data Copies).
  • Immutability protects against a malicious insider and a compromised administrator alike — the two threats that defeat every other backup control.
  • Immutable retention conflicts with data-deletion obligations; resolve it deliberately, usually with crypto-shredding of per-subject keys rather than by weakening retention (Key Management and Encryption at Rest).
  • Restore is a privileged operation that reconstitutes production data somewhere new. It must be audited and should require a second person (Audit Trails).
  • Access to backups must be monitored: an attacker reading your backups is a data breach that leaves the production database untouched.
Cost shape
  • Storage cost is volume × retention, moderated substantially by lifecycle tiering into colder storage.
  • Cross-region and cross-account transfer is metered on every backup written.
  • Restore from cold tiers may carry a retrieval charge and a retrieval delay — check both, because the delay lands inside your RTO.
  • The cheapest layer is usually the most valuable: versioning and soft-delete on object storage recover the most common incidents at almost no cost.
What to watch
  • Backup success, age and size per data set, with alerts on failure, on staleness, and on unexpected size change.
  • Retention compliance: verify that objects genuinely cannot be deleted before expiry rather than trusting the configuration.
  • Backup storage growth as a cost signal and as an early warning of a retention policy that is not applying.
  • Access logs on the backup store, because reads are as interesting as writes here.
  • Date and measured duration of the last successful restore test — the only metric that proves the rest.
  • The signal that lies: a green backup job. It proves a process exited zero, not that the data can come back.
Simpler alternatives
  • Provider-managed point-in-time recovery, if the retention window and account boundary suit your threat model. Simplest possible start and often sufficient for small teams.
  • Object versioning and soft-delete, which recover the most common incident — someone deleted or overwrote the wrong thing — at trivial cost and with no restore procedure.
  • For genuinely derived data, no backup at all. Search indexes, caches and analytics tables that can be recomputed should be documented as deliberately unprotected so the plan stays small.
  • A third-party backup service that holds copies outside your provider entirely, when the threat model includes losing the account.
What adopting this costs
  • Stronger separation costs cross-account complexity and makes restores slightly slower to initiate.
  • Immutability defeats attackers and removes your own ability to delete, which collides with deletion obligations.
  • Longer retention covers late-discovered damage and costs storage linearly.
  • Higher frequency improves RPO and increases load on the source system and the storage bill.
  • Broader coverage means a more complete recovery and more moving parts to keep working.

What people believe, and what is true

Claim

The managed database has automatic backups, so we are covered.

Reality

Those snapshots live in your account, within the provider's retention window, deletable by your administrators, and are lost with the account. They cover hardware failure, which was never the main risk.

Claim

Replication is a backup.

Reality

Replication copies deletions and corruption within seconds. It protects against infrastructure failure and against nothing else on the list.

Claim

A snapshot in another region is enough.

Reality

Only if a compromised production credential cannot delete it. Location is one dimension; deletability is the one attackers exploit.

Apply it