Notification Service
Design a multi-channel notification platform delivering push, email, and SMS at scale using queue-based fanout, per-user rate limiting, retries with backoff, and deduplication.
By Ratish Jain · Updated
Problem
Many services need to notify users through push, email, and SMS without each team re-solving delivery, rate limits, and retries. A central notification service accepts a logical notification, expands it across the user's channels and devices (fanout), and reliably delivers each message via third-party providers (APNs/FCM, an email provider, an SMS gateway).
The hard parts are reliability under provider failures, avoiding spamming users (rate limiting and dedup), and absorbing bursty traffic — a single event can trigger millions of notifications.
Functional Requirements
- Send notifications over push, email, and SMS from a single API.
- Fan out one logical notification to multiple devices/channels per user.
- Respect user preferences, opt-outs, and quiet hours.
- Deduplicate repeated notifications and support templating.
- Track delivery status and expose it to callers.
Non-Functional Requirements
- At-least-once delivery with dedup; no lost notifications.
- High throughput and burst absorption via queuing.
- Per-user and per-channel rate limiting.
- Retry with exponential backoff on transient provider failures.
- Low latency for high-priority (e.g., OTP) messages.
Capacity Estimation
- Users
- 100M active users
- Notifications/day
- 1B (~12K/sec avg, 100K/sec peak)
- Channel mix
- 70% push, 20% email, 10% SMS
- Payload size
- ~1 KB per notification
- Queue throughput
- 100K msgs/sec sustained
- Status store
- 1B rows/day, 7-day TTL ≈ billions rows
APIs
- POST /notifications
- send(userId, template, data, channels) -> {notifId}
- POST /notifications/bulk
- sendBulk(userIds[], template, data) -> {batchId}
- GET /notifications/{id}
- status(notifId) -> {perChannelStatus}
- PUT /preferences
- setPrefs(userId, prefs) -> {status}
Components
- Notify API: validates requests, applies preferences/dedup, enqueues work.
- Message queue (Kafka): durable buffer that decouples producers from workers and absorbs bursts.
- Channel workers: per-channel consumers (push/email/SMS) that format and call providers.
- Rate limiter: Redis-backed counters enforcing per-user/per-channel caps.
- Dedup store: short-TTL key set to drop duplicate notifications.
- Provider adapters: integrations for APNs/FCM, email, and SMS gateways.
- Status store: records per-channel delivery outcomes for querying.
Request Flow
- 1Producer calls the Notify API with a user, template, and target channels.
- 2API checks a dedup key and the user's preferences/opt-outs.
- 3The notification is fanned out into per-channel messages and enqueued.
- 4Channel workers consume messages and enforce rate limits.
- 5Each worker calls its provider; success updates the status store.
- 6Transient failures are retried with backoff; permanent failures are dead-lettered.
Flow Diagram
Data / Storage Design
A template store holds reusable content with variable slots. User preferences/device tokens map a user to channels, devices, and opt-out settings. The dedup store is a Redis set of dedupKey (often a hash of userId + eventId) with a short TTL to collapse duplicate triggers.
Delivery status is high-volume and append-heavy, so it fits a wide-column store (e.g., Cassandra) partitioned by userId or notifId with a TTL. The queue itself (Kafka) is the durable source of truth for in-flight work, letting workers replay after crashes.
Data Model
Scaling Strategy
The queue is the scaling lever: partition topics by userId for ordering per user and add consumers to raise throughput. Each channel scales independently — SMS may be provider-rate-limited while push scales freely.
Separate priority queues (or topics) route OTP/critical traffic ahead of marketing blasts. Bulk sends are expanded asynchronously so a single API call for millions of users doesn't block. Rate-limit counters live in Redis, sharded by user to avoid a hotspot.
Bottlenecks & Failure Scenarios
- Provider throttling or outage — needs backoff, circuit breaking, and dead-letter queues.
- Duplicate delivery if dedup TTL is too short or retries lack idempotency keys.
- Fanout amplification: one bulk event floods the queue and starves priority traffic.
- Rate-limiter hotspot for a viral user or event.
- Poison messages that repeatedly fail and clog a partition.
Design Tradeoffs
- Queue decoupling absorbs bursts and isolates slow providers.
- Per-channel workers scale and fail independently.
- At-least-once + dedup gives reliability without complex exactly-once.
- At-least-once means occasional duplicates must be tolerated downstream.
- Asynchronous flow makes end-to-end latency harder to guarantee.
- Operational overhead of many provider integrations and dead-letter handling.
Common Interview Questions
Callers supply an idempotency/dedup key (or the service hashes userId+eventId). A Redis set with a TTL records seen keys; a repeat within the window is dropped. Workers also key provider calls on the notification id so retries don't double-send.
Failed sends retry with exponential backoff and jitter. A circuit breaker on the adapter stops hammering a dead provider; messages that exhaust retries go to a dead-letter queue for inspection or later replay.
Use separate queues/topics or a priority field so critical, low-latency messages are consumed by dedicated workers ahead of bulk campaigns, preventing head-of-line blocking.
- →Central API fans one notification out across channels and devices.
- →Kafka decouples producers from workers and absorbs bursts.
- →Per-channel workers apply rate limits and call providers.
- →At-least-once delivery + Redis dedup keys prevent spam.
- →Retries with backoff and dead-letter queues handle provider failures.
Related
Everything on Archprimer is connected — follow the graph.