System designIntermediate
Design a Notification System
One service that every other service calls to reach a user by email, SMS or push — with preferences, templates, rate limits, retries and delivery status. It is the cleanest example of queue + workers + external providers, and of why "just call the email API" stops working.
Functional requirements
- Any internal service can request a notification for a user with a type (
order_shipped,otp,weekly_digest) and payload. - Channels: email, SMS, mobile push; a notification can target one or several, chosen by type and user preference.
- User preferences: opt-in/out per channel and per category; quiet hours; per-device push tokens.
- Templates per type and locale, rendered with the payload; content is stored as sent.
- Delivery status per notification and channel: queued, sent, delivered, bounced, failed — queryable by the caller and by support.
- Transactional notifications (OTP, password reset) must go out within seconds; marketing campaigns may take minutes.
- The same event must not produce duplicate notifications when the caller retries.
Non-functional requirements
Scale, latency, availability and durability targets — these decide the architecture.
- Transactional latency: p95 < 5 s from request to provider hand-off; marketing: within 10 minutes of schedule.
- Scale: 100M notifications/day baseline; campaigns of 10M in 10 minutes.
- At-least-once delivery to the provider with deduplication; no notification lost on a crash.
- Availability 99.95% for accepting requests (the API), independent of any provider’s availability.
- Provider limits respected: e.g. email 5,000/s per account, SMS 100/s per sending pool, push 10k/s per app.
- Per-user caps: no more than 5 pushes/hour and 2 marketing emails/day.
Back-of-the-envelope
Numbers first. Every component below has to be justified by one of these.
| Quantity | Value | Arithmetic |
|---|---|---|
| Baseline rate | ≈ 1,160 /s | 100M/day ÷ 86,400 ≈ 1,157/s average; 3× diurnal peak ≈ 3,500/s. A single queue and a few dozen workers. |
| Campaign burst | ≈ 16,700 /s for 10 min | 10M in 10 min = 10,000,000 ÷ 600 s ≈ 16,700/s — 14× baseline. The queue absorbs it; workers drain at provider speed: 10M emails at 5,000/s = 33 min, so "within 10 minutes" needs either a second provider account or a lower promise. |
| Provider drain time | email 5k/s → 10M in 33 min | The provider quota, not our CPU, bounds campaign throughput. Transactional traffic must not queue behind the campaign → separate queues and reserved quota (e.g. 1,000/s held back). |
| Storage | ≈ 50 GB/day, 4.5 TB for 90 days | 100M × ~500 B (ids, status, rendered subject, provider ids, timestamps) = 50 GB/day; 90-day retention ≈ 4.5 TB. Rendered bodies (2–20 KB) go to object storage, not the DB. |
| Dedup keys | ≈ 100M keys × 60 B ≈ 6 GB | One Redis key per notification for 24 h: 100M × ~60 B ≈ 6 GB. Fits one node; shard by key if retention grows. |
| Retry load | ≈ +10–20% | Transient failure rate ~5% per attempt with up to 4 retries adds ~5% + 0.25% + … ≈ 5.3% extra sends; a provider outage of 10 min turns 600 × 1,160 ≈ 700k notifications into retries — the delayed queue must hold that without blocking new work. |
Interface
Endpoints, messages or events.
POST /notifications { user_id, type, payload, idempotency_key, channels?, scheduled_at?, priority? } → 202 { notification_id, status: queued }Callers pass a stable idempotency_key (e.g. order_shipped:{order_id}); a repeat returns the original notification_id. Enqueues after preference resolution; never calls a provider inline.GET /notifications/{id} → { status, channels: { email: { status, attempts, provider_id, last_error }, push: {…} } }Per-channel status; "delivered" for email means the provider reported delivery, not that it was read.PUT /users/{id}/preferences { email: { marketing: false }, push: { all: true }, quiet_hours: { from: "22:00", to: "07:00", tz } } → 200Whole-document PUT, idempotent. Transactional categories (otp, security) cannot be disabled.POST /users/{id}/devices { platform, push_token } → 201 · DELETE /users/{id}/devices/{token}Tokens are per device; provider feedback (unregistered) deletes them automatically.PUT /templates/{type}/{locale} { subject, body_html, body_text, sms, push_title, push_body } → 200Versioned; a notification records the template version it rendered with.POST /webhooks/{provider} (provider → us: delivered, bounced, complained, unregistered)Signed; deduplicated by provider event id; updates channel status and suppression lists (hard bounce → suppress the address).POST /campaigns { type, segment_id, scheduled_at, payload } → 202 { campaign_id }Expands the segment in batches of 10k into per-user notifications on the low-priority queue; progress is reported per campaign.Build it one problem at a time
Each step names the problem first. Decide what you would add before revealing the reference answer.
1
Why every service calling the email API directly fails
Problem · Order service calls SendGrid, Auth service calls Twilio, each with its own retry logic (or none). When the email provider had a 20-minute outage, checkout latency rose to 25 s because the confirmation email was sent inline; nobody honoured the user’s unsubscribe because three services had three preference tables.
Work through every step to unlock the data model, the request walkthrough, scaling, failure modes and the open decisions.