Archprimer
ArchitectureIntermediate 7 min read

Notification Service

Design a multi-channel notification platform delivering push, email, and SMS at scale using queue-based fanout, per-user rate limiting, retries with backoff, and deduplication.

fanoutqueuemulti-channelretriesdedup

By Ratish Jain · Updated

Diagram with 6 components: Producer, Notify API, Message Queue, Push Worker, Email Worker, SMS Worker. Connections: Producer → Notify API (1 · send); Notify API → Message Queue (2 · enqueue); Message Queue → Push Worker (3 · push); Message Queue → Email Worker (3 · email); Message Queue → SMS Worker (3 · sms).
ServiceGatewayQueue
Notifications are enqueued, fanned out per channel, then delivered via providers.

Problem

Many services need to notify users through push, email, and SMS without each team re-solving delivery, rate limits, and retries. A central notification service accepts a logical notification, expands it across the user's channels and devices (fanout), and reliably delivers each message via third-party providers (APNs/FCM, an email provider, an SMS gateway).

The hard parts are reliability under provider failures, avoiding spamming users (rate limiting and dedup), and absorbing bursty traffic — a single event can trigger millions of notifications.

Functional Requirements

  • Send notifications over push, email, and SMS from a single API.
  • Fan out one logical notification to multiple devices/channels per user.
  • Respect user preferences, opt-outs, and quiet hours.
  • Deduplicate repeated notifications and support templating.
  • Track delivery status and expose it to callers.

Non-Functional Requirements

  • At-least-once delivery with dedup; no lost notifications.
  • High throughput and burst absorption via queuing.
  • Per-user and per-channel rate limiting.
  • Retry with exponential backoff on transient provider failures.
  • Low latency for high-priority (e.g., OTP) messages.

Capacity Estimation

Users
100M active users
Notifications/day
1B (~12K/sec avg, 100K/sec peak)
Channel mix
70% push, 20% email, 10% SMS
Payload size
~1 KB per notification
Queue throughput
100K msgs/sec sustained
Status store
1B rows/day, 7-day TTL ≈ billions rows

APIs

POST /notifications
send(userId, template, data, channels) -> {notifId}
POST /notifications/bulk
sendBulk(userIds[], template, data) -> {batchId}
GET /notifications/{id}
status(notifId) -> {perChannelStatus}
PUT /preferences
setPrefs(userId, prefs) -> {status}

Components

  • Notify API: validates requests, applies preferences/dedup, enqueues work.
  • Message queue (Kafka): durable buffer that decouples producers from workers and absorbs bursts.
  • Channel workers: per-channel consumers (push/email/SMS) that format and call providers.
  • Rate limiter: Redis-backed counters enforcing per-user/per-channel caps.
  • Dedup store: short-TTL key set to drop duplicate notifications.
  • Provider adapters: integrations for APNs/FCM, email, and SMS gateways.
  • Status store: records per-channel delivery outcomes for querying.

Request Flow

  1. 1Producer calls the Notify API with a user, template, and target channels.
  2. 2API checks a dedup key and the user's preferences/opt-outs.
  3. 3The notification is fanned out into per-channel messages and enqueued.
  4. 4Channel workers consume messages and enforce rate limits.
  5. 5Each worker calls its provider; success updates the status store.
  6. 6Transient failures are retried with backoff; permanent failures are dead-lettered.

Flow Diagram

Diagram with 6 components: Ingest API, Notify Queue, Fanout Worker, APNs / FCM, Email SMTP, SMS Gateway. Connections: Ingest API → Notify Queue (1 · enqueue); Notify Queue → Fanout Worker (2 · dequeue); Fanout Worker → APNs / FCM (3 · push); Fanout Worker → Email SMTP (3 · email); Fanout Worker → SMS Gateway (3 · sms).
ServiceQueueExternal
Requests are enqueued, fanned out, and pushed to per-channel providers.

Data / Storage Design

A template store holds reusable content with variable slots. User preferences/device tokens map a user to channels, devices, and opt-out settings. The dedup store is a Redis set of dedupKey (often a hash of userId + eventId) with a short TTL to collapse duplicate triggers.

Delivery status is high-volume and append-heavy, so it fits a wide-column store (e.g., Cassandra) partitioned by userId or notifId with a TTL. The queue itself (Kafka) is the durable source of truth for in-flight work, letting workers replay after crashes.

Data Model

Diagram with 5 components: Kafka Topics, Channel Worker, Dedup Store, Status Store, Providers. Connections: Kafka Topics → Channel Worker (1 · consume); Channel Worker → Dedup Store (2 · dedup?); Channel Worker → Status Store (3 · record); Channel Worker → Providers (4 · send); Providers → Status Store (5 · ack).
QueueServiceCacheDatabaseExternal
Workers consume topics, drop duplicates, and record delivery state.

Scaling Strategy

The queue is the scaling lever: partition topics by userId for ordering per user and add consumers to raise throughput. Each channel scales independently — SMS may be provider-rate-limited while push scales freely.

Separate priority queues (or topics) route OTP/critical traffic ahead of marketing blasts. Bulk sends are expanded asynchronously so a single API call for millions of users doesn't block. Rate-limit counters live in Redis, sharded by user to avoid a hotspot.

Bottlenecks & Failure Scenarios

  • Provider throttling or outage — needs backoff, circuit breaking, and dead-letter queues.
  • Duplicate delivery if dedup TTL is too short or retries lack idempotency keys.
  • Fanout amplification: one bulk event floods the queue and starves priority traffic.
  • Rate-limiter hotspot for a viral user or event.
  • Poison messages that repeatedly fail and clog a partition.

Design Tradeoffs

Pros
  • Queue decoupling absorbs bursts and isolates slow providers.
  • Per-channel workers scale and fail independently.
  • At-least-once + dedup gives reliability without complex exactly-once.
Cons
  • At-least-once means occasional duplicates must be tolerated downstream.
  • Asynchronous flow makes end-to-end latency harder to guarantee.
  • Operational overhead of many provider integrations and dead-letter handling.

Common Interview Questions

How do you avoid sending duplicate notifications?

Callers supply an idempotency/dedup key (or the service hashes userId+eventId). A Redis set with a TTL records seen keys; a repeat within the window is dropped. Workers also key provider calls on the notification id so retries don't double-send.

How do you handle a provider outage?

Failed sends retry with exponential backoff and jitter. A circuit breaker on the adapter stops hammering a dead provider; messages that exhaust retries go to a dead-letter queue for inspection or later replay.

How do you prioritize OTP over marketing?

Use separate queues/topics or a priority field so critical, low-latency messages are consumed by dedicated workers ahead of bulk campaigns, preventing head-of-line blocking.

60-Second Revision
  • →Central API fans one notification out across channels and devices.
  • →Kafka decouples producers from workers and absorbs bursts.
  • →Per-channel workers apply rate limits and call providers.
  • →At-least-once delivery + Redis dedup keys prevent spam.
  • →Retries with backoff and dead-letter queues handle provider failures.

Related

Everything on Archprimer is connected — follow the graph.