News Feed
Design a social news feed covering feed generation, fanout-on-write vs fanout-on-read, ranking, and caching of hot feeds for millions of users.
By Ratish Jain · Updated
Problem
A news feed shows each user a personalized, ranked stream of posts from the accounts they follow. At scale the core tension is between write cost and read cost: precompute each user's feed when a post is created (fanout-on-write), or assemble it on demand when they open the app (fanout-on-read). Neither wins universally — a celebrity with 50M followers makes fanout-on-write explode, while an inactive user makes fanout-on-read wasteful.
The design blends both strategies, caches hot feeds aggressively, and ranks posts by relevance rather than pure recency.
Functional Requirements
- Publish posts and deliver them to followers' feeds.
- Return a ranked feed for a user with pagination.
- Support follow/unfollow updating feed composition.
- Rank posts by relevance, not just chronological order.
- Reflect new posts in feeds within seconds.
Non-Functional Requirements
- Feed read latency p99 under ~200ms.
- High availability and read-heavy scalability.
- Eventual consistency is acceptable for feed freshness.
- Handle skewed follower distributions (celebrities).
- Efficient storage — avoid duplicating posts excessively.
Capacity Estimation
- Users
- 500M total, 200M DAU
- Posts/day
- 100M (~1.2K/sec avg)
- Feed reads/day
- 10B (~120K/sec)
- Read:write ratio
- ~100:1 (read-heavy)
- Avg followers
- ~500; celebrities 10M–50M
- Feed cache
- top ~1000 items/user in Redis
APIs
- POST /posts
- createPost(userId, content) -> {postId}
- GET /feed
- getFeed(userId, cursor, limit) -> {items[], next}
- POST /follow
- follow(userId, targetId) -> {status}
- GET /posts/{id}
- getPost(postId) -> {post}
Components
- Feed API: entry point for posting and reading feeds.
- Post store: durable storage of all posts (source of truth).
- Fanout service: on write, pushes new post ids into followers' feed lists.
- Feed cache (Redis): per-user list of recent post ids for fast reads.
- Ranker: scores candidate posts using engagement, recency, affinity.
- Follow/graph store: who-follows-whom, used to compute fanout targets.
- Async workers / queue to process fanout off the write path.
Request Flow
- 1User creates a post; the API writes it to the durable post store.
- 2A fanout job enqueues the post id for the author's followers.
- 3Workers push the id into each follower's cached feed list (fanout-on-write).
- 4For celebrities, the post is skipped and merged at read time (fanout-on-read).
- 5On feed read, the API fetches cached ids and merges any pull-based sources.
- 6The ranker scores and orders items; the page is returned and cached.
Flow Diagram
Data / Storage Design
Posts live in a durable store keyed by postId (source of truth). Each user's feed is a cached list of post ids in Redis (e.g., a sorted set by score/time), holding only the top ~1000 items — hydrated to full posts on read to avoid duplicating post bodies per follower.
The social graph (followers) drives fanout targets. A hybrid model tags high-follower accounts so their posts are not fanned out on write; instead they're pulled and merged when a follower reads. This keeps the write amplification bounded while the vast majority of normal-user posts are precomputed for fast reads.
Data Model
Scaling Strategy
The system is read-heavy, so feeds are precomputed and cached to make reads a cheap cache lookup. Fanout-on-write serves the common case; fanout-on-read handles celebrities to avoid write storms — this hybrid is the key scaling decision.
Fanout runs asynchronously through a queue and a worker pool that scales with post volume. Feed caches are sharded by userId. The graph store is sharded too, and hot celebrity posts are additionally cached so the read-time merge is cheap.
Bottlenecks & Failure Scenarios
- Celebrity fanout amplification — one post writing to tens of millions of feeds.
- Feed cache misses forcing expensive on-read assembly.
- Thundering-herd re-computation when a hot user's cache expires.
- Fanout worker backlog delaying feed freshness.
- Ranking cost growing with candidate-set size.
Design Tradeoffs
- Fanout-on-write makes reads a fast cache lookup (great for read-heavy loads).
- Hybrid model bounds write amplification from celebrities.
- Caching post ids (not bodies) keeps feed storage small.
- Fanout-on-write wastes work for inactive users.
- Hybrid logic adds complexity at read time (merge two sources).
- Eventual consistency means brief feed staleness.
Common Interview Questions
On-write precomputes each follower's feed when a post is made — fast reads, but huge write amplification for high-follower accounts. On-read assembles the feed at query time — cheap writes, slower reads. Production systems use a hybrid: on-write for normal users, on-read for celebrities.
Skip fanout-on-write for them; store their posts once and merge them into a follower's feed at read time. This caps write amplification, and their posts are cached so the read-time merge stays cheap.
Rather than pure recency, candidates are scored by signals like author affinity, engagement (likes/comments), recency decay, and content type. The ranker orders the merged candidate set and the top page is cached.
- →Feed = ranked stream of followed accounts' posts; system is read-heavy.
- →Fanout-on-write precomputes feeds for fast reads; costly for celebrities.
- →Fanout-on-read assembles at query time; used for high-follower accounts.
- →Hybrid model + Redis-cached post-id lists is the scaling core.
- →Rank by affinity/engagement/recency, not just chronology.
Related
Everything on Archprimer is connected — follow the graph.