FilesGENERALCLOUD-SPECIFICSIMPLIFIED

Object Storage

A bucket holds objects addressed by a key. That primitive — not any provider's SDK — is what you are actually programming against.

What actually happensHow to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What is the storage primitive underneath every cloud file API, and how does it differ from the filesystem you are picturing?

The requirement

The application needs somewhere durable to put user-uploaded files: cheap, effectively unlimited, reachable from several instances, and still there after any single machine is gone.

The obvious build

Write the files to local disk under /var/app/uploads and serve them with the web server. It is a filesystem — it is the thing files go on — and it works perfectly in development.

Why it breaks

A second instance cannot see the first instance's files. The upload succeeds, the download 404s, and which one you get depends on which instance the load balancer picked (Making an Existing Service Stateless).

How it breaks in production
  • A second instance cannot see the first instance's files. The upload succeeds, the download 404s, and which one you get depends on which instance the load balancer picked (Making an Existing Service Stateless).
  • The instance is replaced on every deploy and by every autoscaling event. Local disk is process state, not application state, and it disappears without notice (Deployment Models).
  • Disk is a fixed size that fills. There is no graceful degradation: writes start failing, and so does anything else on that volume, including logs.
  • Backup, durability and replication become your problem, on a volume nobody set out to make durable.
  • The files are served by the same process that serves your API, so a large download competes with request handling for bandwidth and connections.
RequirementAPI ContractApplication LogicData AccessExternal DepsConcurrencyFailureSecurityObservabilityDeploymentScale

What is actually happening

  • The primitive has three parts and nothing else. A bucket is a named container. An object key is a string that names one object inside that bucket. An object is an immutable blob of bytes plus metadata — content type, size, an entity tag, and whatever custom key-value pairs you attach.
  • The key is a flat string. a/b/c.png contains slashes and this creates no directories: there is no a, no b, and nothing to traverse. Listing with a prefix and a delimiter is what produces the illusion of folders, and it is a listing convention, not a hierarchy (Files, Paths and Names describes the filesystem model this is not).
  • Objects are immutable. There is no seek, no append, no partial write. Changing one byte means uploading a whole new object, usually under the same key, which replaces the old one entirely.
  • The interface is HTTP verbs against keys: PUT to write, GET to read, HEAD for metadata, DELETE to remove, LIST to enumerate by prefix. Every operation is a network request with a latency and a per-request charge, which is why a thousand small objects behaves very differently from one large one.
  • Durability is achieved by replication across machines and usually across failure domains inside a region, which is why object stores quote extremely high durability and much lower availability — the bytes survive, the endpoint may not answer right now (Failure Domains).
  • Storage classes trade retrieval latency and per-request cost against per-byte cost. The cheap tiers are cheap because retrieval is slow, expensive, or both, and because they charge for early deletion (Storage Lifecycle: Hot, Warm, Archive, Delete).
  • Everything about serving is separate from everything about storing: whether an object is public, how it is cached, and who may read it are all decisions layered on top of the primitive (Serving Files).

Bucket, key, object — and nothing else

Start with the primitive, because every provider's SDK is a wrapper over the same three concepts, and the mistakes people make come from importing a filesystem model rather than from the API.

A bucket is a namespace. A key is a string. An object is bytes plus metadata, written whole and replaced whole. The slashes in t/acme/tickets/91/6f2a…png are characters in a string that list operations know how to group by. Nothing is nested; nothing can be moved without being copied.

  • Flat — no directories exist; prefixes are a listing convention.
  • Immutable — no append, no partial write, no seek. Replace the whole object.
  • HTTP — PUT / GET / HEAD / DELETE / LIST, each a network request with a price.
  • Metadata-carrying — content type and custom pairs travel with the object.
  • Durable, not always available — the bytes survive; the endpoint can still fail.
contains keysnames exactly onecarriesenumerate by prefixBucket a flat namespaceKey "t/acme/tickets/91/6f2a.png" one opaque stringLIST prefix="t/acme/" grouping, not traversalObject immutable bytesMetadata type, size, etag, custom
UserLLMAgentToolDataDecisionHumanGuardrail

What a filesystem does that this does not

SIMPLIFIEDSome providers layer filesystem-like semantics on top — GCS has a hierarchical namespace option, Azure Data Lake Storage Gen2 adds real directories and atomic rename, and various FUSE mounts pretend. Those are additional products with their own trade-offs, not properties of the underlying object model.

Most object-storage bugs are a filesystem habit meeting a key-value store. This is the list worth internalizing before writing the first put.

OperationFilesystemObject storage
Address a filePath traversed through directoriesOne flat key string
Create a directoryA real operationDoes not exist; prefixes appear when objects do
Rename / moveCheap metadata change, atomicCopy then delete: O(size), two requests, not atomic
Append to a fileSupportedNot supported; rewrite the whole object
Partial writeSeek and writeNot supported (multipart is for upload, not editing)
Read part of a fileSeek and readRange GET — supported, and still a network request
Lock a fileAdvisory or mandatory locksNo locking; concurrent PUTs are last-write-wins
List a directoryFast, localPaginated LIST, lexicographic, slower as the bucket grows
Delete a directoryOne recursive callList every key, delete each (batched at best)
Cost of an operationA syscallA billed HTTP request with network latency (Everything Is I/O)

The key is a schema decision

Because there is no hierarchy, the key is the only structure you get, and it is effectively permanent — changing a key scheme means copying every object. Three things belong in it: an isolation boundary, a stable path to the owning entity, and an unguessable component.

Note what is deliberately absent: nothing user-supplied, and nothing you would need to change. Display names, revisions and statuses belong in the database, where they can be updated without moving bytes.

A key scheme, and what each segment is for
1// t/{tenant}/{entity}/{entityId}/{uuid}[.ext]
2// | | | | |
3// | | | | +-- unguessable; also prevents overwrite races
4// | | | +------------ ties the object to a row you can authorize
5// | | +--------------------- stable entity type; survives feature renames
6// | +------------------------------- isolation boundary: prefix == tenant
7// +------------------------------------- version the whole scheme, so v2 can coexist
8
9function attachmentKey(ctx: TenantContext, ticketId: string, ext: string) {
10 return `t/${ctx.tenantId}/tickets/${ticketId}/${randomUUID()}${ext}`
11}
12
13// The database owns everything queryable. The bucket owns bytes.
14// attachments(id, tenant_id, ticket_id, key, display_name, content_type,
15// size_bytes, status, created_at)
16//
17// Never: LIST the bucket to find a user's files. That is a scan whose cost
18// grows with every object anyone has ever uploaded.

The tenant-first prefix is what lets an IAM policy or a signed URL be scoped to one customer's data, so the key layout is an authorization mechanism and not only an organizational one (Tenant Isolation). The UUID means two concurrent writes can never target the same key, which removes the last-write-wins race entirely.

How to build it

Most important first.

  • Design the key like a schema, because it is one. Put the tenant first (t/{tenant}/…) so a prefix is an isolation boundary, then a stable entity path, then an unguessable id (Tenant Isolation).
  • Never let a client choose a key. Generate it, store it in your database, and treat the database as the index — the object store is a poor place to look things up.
  • Keep the bucket private by default and grant reads through signed URLs or a CDN, rather than making objects public and relying on unguessable keys (Presigned URLs).
  • Store metadata you will query in your database, and metadata that describes the bytes on the object. Listing a bucket to find something is a scan, and it gets slower with every object.
  • Use lifecycle rules for expiry and tiering — abandoned uploads, old versions, temporary exports — rather than writing a deletion job you will forget to run (Storage Lifecycle: Hot, Warm, Archive, Delete).
  • Enable versioning where accidental overwrite or deletion would be expensive, and understand that versions cost money until a lifecycle rule removes them.
  • Wrap the provider behind a small internal interface (put, get, signPut, signGet, delete) so that provider specifics live in one adapter, while accepting that the abstraction leaks exactly where the providers differ.

What can go wrong

Failure modes
  • Treating the key prefix as a directory and expecting rename or move to be cheap: both are copy-then-delete, proportional to object size, and not atomic.
  • Listing a large prefix in a request path. Listing is paginated, ordered lexicographically, and slow enough to time out on a big bucket.
  • A thousand tiny objects where one archive would do: per-request cost and per-request latency dominate, and the storage bytes are the small part of the bill.
  • Deleting an object that something else still references, because the database row and the object have no referential integrity between them.
  • Assuming a read immediately after a write returns the new value. Modern major providers offer strong read-after-write consistency for objects, but list operations and cross-region replicas have their own semantics — and older systems and some S3-compatible implementations do not make the same guarantee (Eventual Consistency in Practice).
  • An overwrite losing data because two writers used the same key with no versioning and no conditional write.
  • Egress charges discovered after the fact, because serving directly from the bucket to end users bypassed the CDN (Egress: Moving Data Costs Money, Not Just Storing It).
  • Credentials with bucket-wide or account-wide permissions embedded in the application, so a compromise of the app is a compromise of all stored data (Least Privilege in Infrastructure).
What can race
  • Two writers PUT the same key: last write wins, silently, with no error. Conditional writes or unique generated keys are the defences, and support for conditional writes differs by provider.
  • A reader listing a prefix while another writer adds objects sees a snapshot that may not include the new ones; list consistency is weaker than object read consistency on several providers.
  • A lifecycle rule deleting an object between your database read and your client's fetch of a signed URL (Storage Lifecycle: Hot, Warm, Archive, Delete).
Security
  • If the bucket is public, an attacker gets everything in it by enumerating keys — and public buckets are enumerable by design. Unguessable keys reduce discovery and are not access control (Public Exposure, Read With Context).
  • If keys are not tenant-prefixed and access is not scoped by prefix, an attacker who can influence a key gets cross-tenant read or write (Tenant Isolation).
  • If the application's storage credential grants more than the prefixes it needs, an attacker who compromises the application gets read and write across every customer's files rather than the ones that request touched (Anatomy of a Policy).
  • If objects are served from your primary origin, an uploaded HTML or SVG file executes with your origin's privileges — the standard stored-XSS path for file features (Serving Files).
  • If deletion is not verified, an attacker (or a compliance auditor) discovers that "deleted" data still exists as an old version or in a replica bucket, because versioning and replication both keep copies.
  • If encryption at rest is assumed rather than configured, sensitive data may be stored under provider-managed keys when the requirement was customer-managed keys — a difference that matters legally more than technically (Key Management and Encryption at Rest).
Misreads
  • "It is a filesystem in the cloud." It is a key-value store for blobs with an HTTP API. The path-like keys are the most misleading part of the interface (Files, Paths and Names).
  • "Prefixes are folders." They are string prefixes used by list operations. Deleting a prefix means listing and deleting every object under it, one request at a time.
  • "S3, GCS and Azure Blob are the same service with different names." They share the bucket/key/object primitive and differ in consistency documentation, signing schemes, event delivery, access-control models, storage-class semantics and minimum-duration charges. Code written against one migrates with real work, not a config change (Mapping Services Across Cloud Providers).
  • "Storage is cheap, so this is cheap." Storage is cheap. Requests, egress, early deletion from cold tiers and retained versions are the parts of the bill people are surprised by (Cost Engineering).
  • "Deleted means gone." With versioning or replication enabled, deletion creates a marker or leaves copies. Verify against the requirement you actually have.

Operating it

How you see it in production
  • Track request counts by operation, not just stored bytes. A bill dominated by GET and LIST requests is an access-pattern problem, not a storage problem (Cost per Request: The Other Performance Metric).
  • Alert on 4xx and 5xx rates from the storage client separately from your own error rate — storage errors surface as failed user actions with no application bug (An Error Taxonomy That Maps Cause to Response).
  • Measure storage operation latency inside your traces. A slow put inside a request handler is invisible in endpoint-level metrics until it is the whole request (Tracing From the Backend's Side).
  • Reconcile object counts and total size against your database's view on a schedule. Divergence is orphans in one direction and broken references in the other.
  • Watch egress as a first-class metric. It is usually the largest and least expected line on a storage bill (Egress: Moving Data Costs Money, Not Just Storing It).
What changes at 10x and 100x
  • Object stores absorb scale in bytes and object count without your involvement; that is the primary reason to use one. Ten objects and ten billion objects use the same API.
  • What does not scale is *your* access pattern. Listing gets slower, per-request costs grow linearly with request count, and a hot key can hit per-prefix request-rate limits.
  • At high volume, key design starts to matter for throughput: providers partition by key prefix, so keys sharing a long common prefix can concentrate load. Providers differ in how much of this they handle automatically now, and in whether they document it at all.
  • Cost shifts from storage to requests and egress as object counts rise. Serving through a CDN changes the egress number more than any storage-side decision does (CDN as Infrastructure).
What this costs
  • You give up filesystem semantics entirely: no append, no partial write, no atomic rename, no locking, no cheap move. Code that assumed a filesystem needs rethinking, not porting (File Systems: From Path to Blocks).
  • Every operation is a network call. Latency is milliseconds rather than microseconds, and failure is a normal outcome that needs timeouts and retries (Timeouts).
  • Cheap per byte, not per request. Workloads with many tiny objects pay far more than their stored size suggests.
  • Provider coupling is real. The bucket/key/object model transfers; signing, events, consistency guarantees, lifecycle configuration and storage classes do not transfer cleanly.
  • Strong durability does not mean strong availability. You still need a plan for the minutes when the storage endpoint is returning 503s (Circuit Breakers).

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

  • GENERALThe bucket / key / object model with immutable objects, prefix listing and an HTTP verb interface is common to every implementation, including self-hosted ones like MinIO and Ceph RGW.
  • CLOUD-SPECIFICEverything above the primitive differs and the differences reach your code. Access control: S3 uses IAM policies plus bucket policies and (legacy) ACLs; GCS uses IAM with uniform bucket-level access; Azure uses RBAC plus shared access signatures. Events: S3 notifies SNS/SQS/Lambda, GCS publishes to Pub/Sub, Azure uses Event Grid — different payload shapes, different delivery guarantees. Naming: buckets are globally unique in S3 and GCS, while Azure nests containers inside a storage account. Storage classes have different names, different minimum storage durations and different retrieval charges. Treat the primitive as portable and every one of these as a rewrite.
  • SIMPLIFIEDMultipart upload, byte-range reads, conditional requests, object locks and cross-region replication all exist and are omitted here. This lesson teaches the model you program against, not the full API surface.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.