Archprimer
ArchitectureAdvanced 5 min read

Instagram Feed

A photo-sharing service with a personalized timeline, balancing fanout-on-write and fanout-on-read to generate feeds cheaply, and a CDN to serve media globally.

FanoutCDNCachingTimelineSharding

By Ratish Jain · Updated

Diagram with 7 components: App, API Gateway, Feed Service, Fanout Worker, Feed Cache, Post DB, CDN. Connections: App → API Gateway (1 · GET feed); API Gateway → Feed Service (2 · build); Feed Service → Feed Cache (3 · feed ids); Feed Service → Post DB (4 · hydrate); Fanout Worker → Feed Cache (push post); App → CDN (load media).
ClientGatewayServiceCacheDatabaseCDN
Writes fan out into per-user feed caches; reads merge cached feed IDs with hydrated post data; media comes from the CDN.

Problem

Design Instagram: users post photos and see a personalized feed of recent posts from accounts they follow.

The core challenge is the feed generation problem. With hundreds of millions of users and skewed follower graphs (celebrities with tens of millions of followers), naively rebuilding every feed on read or pushing every post to every follower both break. The winning design blends fanout-on-write for normal users with fanout-on-read for celebrities.

Functional Requirements

  • Upload photos with captions; follow/unfollow accounts.
  • View a reverse-chronological (or ranked) feed of followed accounts' posts.
  • Like and comment on posts.
  • Feed loads fast and paginates infinitely.
  • Media served at multiple resolutions.

Non-Functional Requirements

  • Feed read latency under ~200 ms (p99).
  • Read-heavy: ~100:1 read-to-write ratio.
  • High availability; eventual consistency for feed acceptable.
  • Media delivered globally with low latency via CDN.
  • Elastic to handle viral posts and celebrity accounts.

Capacity Estimation

Daily active users
~500M
Posts/day
~100M → ~1,200 posts/s
Feed reads/day
~10B → ~115K reads/s
Avg followers
~200; celebrities up to 100M+
Media/day
100M photos × 2 MB ≈ 200 TB/day
Feed cache
500M users × ~500 ids × 16B ≈ 4 TB

APIs

POST /posts
createPost(userId, mediaId, caption) → postId
GET /feed
getFeed(userId, cursor, limit) → Post[]
POST /follow
follow(userId, targetId) → 200
POST /media
uploadMedia(userId, bytes) → mediaId + urls

Components

  • API gateway: auth, rate limiting, routing.
  • Post service + Post DB: sharded store of post metadata (author, caption, media ref, timestamp).
  • Fanout workers: on new post, push the postId into followers' feed caches (for normal users).
  • Feed cache (Redis): per-user list of recent post IDs — the precomputed timeline.
  • Feed service: assembles a feed by merging cached IDs (fanout-on-write) with pulled celebrity posts (fanout-on-read), then hydrates.
  • Media pipeline + CDN: resizes uploads to multiple resolutions and serves them from edge caches.

Request Flow

  1. 1User uploads a photo; media pipeline stores multi-resolution variants in blob storage and returns a mediaId.
  2. 2Post service writes post metadata to the sharded Post DB.
  3. 3Fanout workers append the postId to each follower's Redis feed list (skipping celebrity followers).
  4. 4On feed read, the Feed service reads the user's cached post IDs.
  5. 5It also pulls recent posts from the celebrities the user follows (fanout-on-read) and merges by timestamp.
  6. 6Post metadata is hydrated from cache/DB; the client loads media directly from the CDN.

Flow Diagram

Diagram with 5 components: Client, Feed API, Feed Cache, Celeb Fanout, Merge. Connections: Client → Feed API (1 · GET feed); Feed API → Feed Cache (2 · read); Feed API → Celeb Fanout (3 · pull); Feed Cache → Merge (4 · merge); Celeb Fanout → Merge.
ClientGatewayCacheService
Feed read: cache hit merged with celebrity pull-on-read.

Data / Storage Design

Post metadata lives in a sharded store keyed by postId (or userId), separate from media. The social graph (followers) is stored in its own service, sharded by userId, since follower lists are read on every fanout.

The feed cache is the crux: a Redis list per user holding the newest ~hundreds of post IDs. It is a materialized view, cheaply rebuildable from posts + graph, so it can be evicted for inactive users to save memory. Media bytes live in blob storage fronted by a CDN; only immutable URLs referencing a content hash flow through the app tier.

Data Model

Diagram with 6 components: Feed Svc, Feed Cache, Media Blob, Post Shard A, Post Shard B, CDN. Connections: Feed Svc → Feed Cache (1 · feed); Feed Svc → Post Shard A (2 · posts); Feed Svc → Post Shard B; Feed Svc → Media Blob (3 · media); Media Blob → CDN (4 · serve).
ServiceCacheStorageDatabaseCDN
Sharded posts + feed cache; media served from blob via CDN.

Scaling Strategy

The hybrid fanout is the scaling lever. Fanout-on-write precomputes feeds so reads are cheap — great for the 99% of users with modest follower counts. But pushing a celebrity's post to 100M feeds is wasteful, so those are handled by fanout-on-read: the reader pulls the celebrity's recent posts at feed-build time and merges them in.

Feed caches, post DB, and the graph service all shard by user/post ID. The CDN offloads virtually all media bandwidth from origin. Inactive users' feed caches are lazily rebuilt to bound Redis memory.

Bottlenecks & Failure Scenarios

  • Celebrity fanout storm: writing to tens of millions of caches — solved by fanout-on-read for high-follower accounts.
  • Redis feed cache eviction/loss: rebuild feed lazily from posts + graph; degrade to fanout-on-read.
  • Hot shard: a viral post's counters (likes) hammer one shard — use sharded counters / write-back aggregation.
  • Thundering herd on cache miss: request coalescing and cache-aside with jittered TTLs.
  • Media origin overload: absorbed by CDN edge caching; origin sees only cache fills.

Design Tradeoffs

Pros
  • Precomputed feeds make the read path (the hot path) extremely fast.
  • Hybrid fanout avoids the celebrity write-amplification trap.
  • CDN + immutable media URLs slash bandwidth and latency.
Cons
  • Feed cache storage is large and must be actively managed.
  • Eventual consistency: a new post may take seconds to appear in all feeds.
  • Merging two fanout strategies at read time adds code complexity.

Common Interview Questions

Fanout-on-write vs fanout-on-read — when do you use each?

Fanout-on-write precomputes feeds and is ideal for normal users (cheap reads). Fanout-on-read pulls at query time and suits celebrities, where pushing to millions of caches would be prohibitive. Instagram uses a hybrid keyed on follower count.

How do you keep the feed cache from exploding in memory?

Cap each user's list length, and only maintain caches for active users. Inactive users' feeds are evicted and rebuilt lazily on next login from posts + graph.

How is media served efficiently?

Photos are pre-resized into fixed resolutions, stored in blob storage with content-hash URLs, and served from a CDN so 99%+ of requests never reach origin.

60-Second Revision
  • →Feed = precomputed Redis list of post IDs per user.
  • →Fanout-on-write for normal users; fanout-on-read for celebrities.
  • →Post metadata sharded and separate from media bytes.
  • →CDN serves multi-resolution media via immutable URLs.
  • →Feed cache is a rebuildable materialized view — evict freely.

Related

Everything on Archprimer is connected — follow the graph.