Archprimer
ArchitectureAdvanced 8 min read

News Feed

Design a social news feed covering feed generation, fanout-on-write vs fanout-on-read, ranking, and caching of hot feeds for millions of users.

fanoutfeed-generationrankingcachingtimeline

By Ratish Jain · Updated

Diagram with 6 components: Client, Feed API, Fanout Svc, Feed Cache, Post Store, Ranker. Connections: Client → Feed API (1 · read/post); Feed API → Post Store (2 · write); Post Store → Fanout Svc (3 · fanout); Fanout Svc → Feed Cache (4 · push); Feed API → Feed Cache (5 · fetch); Feed Cache → Ranker (6 · rank).
ClientGatewayServiceCacheDatabase
Post fans out to follower feeds; reads served from cached timelines.

Problem

A news feed shows each user a personalized, ranked stream of posts from the accounts they follow. At scale the core tension is between write cost and read cost: precompute each user's feed when a post is created (fanout-on-write), or assemble it on demand when they open the app (fanout-on-read). Neither wins universally — a celebrity with 50M followers makes fanout-on-write explode, while an inactive user makes fanout-on-read wasteful.

The design blends both strategies, caches hot feeds aggressively, and ranks posts by relevance rather than pure recency.

Functional Requirements

  • Publish posts and deliver them to followers' feeds.
  • Return a ranked feed for a user with pagination.
  • Support follow/unfollow updating feed composition.
  • Rank posts by relevance, not just chronological order.
  • Reflect new posts in feeds within seconds.

Non-Functional Requirements

  • Feed read latency p99 under ~200ms.
  • High availability and read-heavy scalability.
  • Eventual consistency is acceptable for feed freshness.
  • Handle skewed follower distributions (celebrities).
  • Efficient storage — avoid duplicating posts excessively.

Capacity Estimation

Users
500M total, 200M DAU
Posts/day
100M (~1.2K/sec avg)
Feed reads/day
10B (~120K/sec)
Read:write ratio
~100:1 (read-heavy)
Avg followers
~500; celebrities 10M–50M
Feed cache
top ~1000 items/user in Redis

APIs

POST /posts
createPost(userId, content) -> {postId}
GET /feed
getFeed(userId, cursor, limit) -> {items[], next}
POST /follow
follow(userId, targetId) -> {status}
GET /posts/{id}
getPost(postId) -> {post}

Components

  • Feed API: entry point for posting and reading feeds.
  • Post store: durable storage of all posts (source of truth).
  • Fanout service: on write, pushes new post ids into followers' feed lists.
  • Feed cache (Redis): per-user list of recent post ids for fast reads.
  • Ranker: scores candidate posts using engagement, recency, affinity.
  • Follow/graph store: who-follows-whom, used to compute fanout targets.
  • Async workers / queue to process fanout off the write path.

Request Flow

  1. 1User creates a post; the API writes it to the durable post store.
  2. 2A fanout job enqueues the post id for the author's followers.
  3. 3Workers push the id into each follower's cached feed list (fanout-on-write).
  4. 4For celebrities, the post is skipped and merged at read time (fanout-on-read).
  5. 5On feed read, the API fetches cached ids and merges any pull-based sources.
  6. 6The ranker scores and orders items; the page is returned and cached.

Flow Diagram

Diagram with 6 components: Author, Post API, Post Store, Fanout Worker, Feed Cache, Reader. Connections: Author → Post API (1 · post); Post API → Post Store (2 · store); Post API → Fanout Worker (3 · fanout); Fanout Worker → Feed Cache (4 · push); Reader → Feed Cache (5 · read).
ClientServiceDatabaseCache
A new post is persisted, fanned out to follower feeds, and read from cache.

Data / Storage Design

Posts live in a durable store keyed by postId (source of truth). Each user's feed is a cached list of post ids in Redis (e.g., a sorted set by score/time), holding only the top ~1000 items — hydrated to full posts on read to avoid duplicating post bodies per follower.

The social graph (followers) drives fanout targets. A hybrid model tags high-follower accounts so their posts are not fanned out on write; instead they're pulled and merged when a follower reads. This keeps the write amplification bounded while the vast majority of normal-user posts are precomputed for fast reads.

Data Model

Diagram with 5 components: Fanout Worker, Graph Store, Post Store, Feed Cache, Reader. Connections: Fanout Worker → Graph Store (1 · followers); Fanout Worker → Post Store (2 · fetch); Fanout Worker → Feed Cache (3 · fanout); Reader → Feed Cache (4 · read).
ServiceDatabaseCacheClient
Fanout reads the follow graph, pulls posts, materializes per-user feeds.

Scaling Strategy

The system is read-heavy, so feeds are precomputed and cached to make reads a cheap cache lookup. Fanout-on-write serves the common case; fanout-on-read handles celebrities to avoid write storms — this hybrid is the key scaling decision.

Fanout runs asynchronously through a queue and a worker pool that scales with post volume. Feed caches are sharded by userId. The graph store is sharded too, and hot celebrity posts are additionally cached so the read-time merge is cheap.

Bottlenecks & Failure Scenarios

  • Celebrity fanout amplification — one post writing to tens of millions of feeds.
  • Feed cache misses forcing expensive on-read assembly.
  • Thundering-herd re-computation when a hot user's cache expires.
  • Fanout worker backlog delaying feed freshness.
  • Ranking cost growing with candidate-set size.

Design Tradeoffs

Pros
  • Fanout-on-write makes reads a fast cache lookup (great for read-heavy loads).
  • Hybrid model bounds write amplification from celebrities.
  • Caching post ids (not bodies) keeps feed storage small.
Cons
  • Fanout-on-write wastes work for inactive users.
  • Hybrid logic adds complexity at read time (merge two sources).
  • Eventual consistency means brief feed staleness.

Common Interview Questions

Fanout-on-write vs fanout-on-read?

On-write precomputes each follower's feed when a post is made — fast reads, but huge write amplification for high-follower accounts. On-read assembles the feed at query time — cheap writes, slower reads. Production systems use a hybrid: on-write for normal users, on-read for celebrities.

How do you handle a celebrity with 50M followers?

Skip fanout-on-write for them; store their posts once and merge them into a follower's feed at read time. This caps write amplification, and their posts are cached so the read-time merge stays cheap.

How is the feed ranked?

Rather than pure recency, candidates are scored by signals like author affinity, engagement (likes/comments), recency decay, and content type. The ranker orders the merged candidate set and the top page is cached.

60-Second Revision
  • →Feed = ranked stream of followed accounts' posts; system is read-heavy.
  • →Fanout-on-write precomputes feeds for fast reads; costly for celebrities.
  • →Fanout-on-read assembles at query time; used for high-follower accounts.
  • →Hybrid model + Redis-cached post-id lists is the scaling core.
  • →Rank by affinity/engagement/recency, not just chronology.

Related

Everything on Archprimer is connected — follow the graph.