System designIntermediate
Design a File Storage Service
Upload, store, share and download files of any size — a Dropbox-style service. The design separates two very different kinds of data (small hot metadata, huge cold bytes), moves bytes off the API servers with presigned URLs, and handles the realities of 2 GB uploads on flaky networks, duplicate content and malware.
Functional requirements
- Upload files up to 5 GB from web and mobile; an interrupted upload resumes rather than restarting.
- Folders, rename, move, delete (with a 30-day trash), and listing a folder with sorting and paging.
- Download at full speed from anywhere; previews/thumbnails for images and documents.
- Share a file or folder with specific users (viewer/editor) or via a link (optional password and expiry).
- Identical files uploaded by different users should not be stored twice.
- Every uploaded file is scanned for malware before it can be downloaded by others.
- Per-user storage quota enforced.
Non-functional requirements
Scale, latency, availability and durability targets — these decide the architecture.
- Durability 99.999999999% (11 nines) for stored bytes — a file, once acknowledged, is never lost.
- Metadata operations (list, rename, share) p99 < 100 ms; download time-to-first-byte p99 < 200 ms from a CDN edge.
- Scale: 10M uploads/day averaging 1 MB (1% over 100 MB), 50M downloads/day, 50M users, 10 PB stored.
- Availability 99.95% for metadata and download; upload can degrade to "retry later".
- Upload throughput per client limited only by the client’s bandwidth, not by our API servers.
Back-of-the-envelope
Numbers first. Every component below has to be justified by one of these.
| Quantity | Value | Arithmetic |
|---|---|---|
| Ingest | ≈ 10 TB/day, 116 uploads/s | 10M uploads/day ÷ 86,400 ≈ 116/s, peak 3× ≈ 350/s; × 1 MB average = 116 MB/s ≈ 0.9 Gbit/s average, ~3 Gbit/s peak. Through API servers at 1 Gbit/s each that is 3+ servers doing nothing but relaying bytes — the reason for direct-to-storage uploads. |
| Egress | ≈ 5 Gbit/s, 70% from CDN | 50M downloads/day ≈ 580/s × 1 MB ≈ 580 MB/s ≈ 4.6 Gbit/s. With ~70% CDN hit rate (popular shared files), origin serves ~1.4 Gbit/s. |
| Storage growth | ≈ 3.65 PB/yr, ~2.5 PB after dedup | 10 TB/day × 365 = 3.65 PB/yr; content-hash dedup typically saves 25–35% (installers, shared documents) → ~2.5 PB/yr net. Object storage at ~$0.02/GB-month ≈ $50k/month per 2.5 PB. |
| Metadata | ≈ 5 GB/day, fits Postgres for years | 10M files/day × ~500 B (ids, name, size, hash, parent, owner, timestamps) = 5 GB/day, 1.8 TB/yr. Postgres with partitioning by owner range or a sharded setup handles it; the metadata is small relative to the bytes. |
| Chunks | 4 MB → a 2 GB file = 512 parts | A 5 GB maximum at 4 MB chunks is 1,250 parts; multipart limits (10,000 parts) are far away. 4 MB balances retry cost (a lost chunk = 4 MB re-sent) against per-part overhead (one signed URL and one ETag each). |
| Scan load | ≈ 116 scans/s × ~1 s | Scanning 116 files/s at ~1 s each = ~120 concurrent scanners; large files take longer. A queue with an autoscaled worker pool sized on depth, not a synchronous step in the upload. |
Interface
Endpoints, messages or events.
POST /uploads { name, size, sha256, parent_id, mime } → 201 { upload_id, chunk_size, parts: [{ n, url }] } | 200 { file_id, deduplicated: true }Client hashes the file first. If a blob with that sha256 exists in the caller’s account (or globally, see the dedup decision), the file is created instantly. Otherwise presigned PUT URLs for each 4 MB part, valid 1 h; upload_id valid 24 h.PUT <presigned part url> (client → object storage)Bytes never touch our servers. The signature binds bucket, key, part number and expiry; storage returns an ETag per part. Retry any part independently.POST /uploads/{id}/complete { parts: [{ n, etag }] } → 201 { file_id, status: scanning }Completes the multipart object, verifies size and (optionally) the hash, creates the files row, refcounts the blob, enqueues the scan. Idempotent: completing twice returns the same file_id.GET /uploads/{id} → { received_parts: [1,2,5…] }Resume: the client asks which parts landed and uploads only the missing ones.GET /files/{id}/download → 302 Location: <signed CDN url, 1 h>Authorises via the ACL, then redirects to a signed CDN URL keyed by blob hash so identical files share cache entries. Blocked (403) while status = scanning/quarantined for anyone but the owner.GET /folders/{id}/children?sort=name&cursor=&limit=100 → { items[], next_cursor }Keyset paging on (name, id); metadata only, no storage calls.POST /files/{id}/shares { principal: user_id | "link", role: viewer|editor, expires_at?, password? } → 201 { share_id, url? }Creates an ACL entry or a link token. Folder shares apply to all descendants via inheritance resolved at check time.DELETE /files/{id} → 204 · POST /files/{id}/restoreSoft delete (trashed_at); a purge job after 30 days decrements the blob refcount and deletes the object when it reaches zero.Build it one problem at a time
Each step names the problem first. Decide what you would add before revealing the reference answer.
1
Separate metadata from bytes
Problem · Version one streams uploads through the API server to a local disk and stores the path in a table. The API server’s 1 Gbit/s NIC saturates at ~120 MB/s of uploads, one disk failure loses files, and "list my folder" competes for I/O with a 2 GB upload.
Work through every step to unlock the data model, the request walkthrough, scaling, failure modes and the open decisions.