Instagram Feed
A photo-sharing service with a personalized timeline, balancing fanout-on-write and fanout-on-read to generate feeds cheaply, and a CDN to serve media globally.
By Ratish Jain · Updated
Problem
Design Instagram: users post photos and see a personalized feed of recent posts from accounts they follow.
The core challenge is the feed generation problem. With hundreds of millions of users and skewed follower graphs (celebrities with tens of millions of followers), naively rebuilding every feed on read or pushing every post to every follower both break. The winning design blends fanout-on-write for normal users with fanout-on-read for celebrities.
Functional Requirements
- Upload photos with captions; follow/unfollow accounts.
- View a reverse-chronological (or ranked) feed of followed accounts' posts.
- Like and comment on posts.
- Feed loads fast and paginates infinitely.
- Media served at multiple resolutions.
Non-Functional Requirements
- Feed read latency under ~200 ms (p99).
- Read-heavy: ~100:1 read-to-write ratio.
- High availability; eventual consistency for feed acceptable.
- Media delivered globally with low latency via CDN.
- Elastic to handle viral posts and celebrity accounts.
Capacity Estimation
- Daily active users
- ~500M
- Posts/day
- ~100M → ~1,200 posts/s
- Feed reads/day
- ~10B → ~115K reads/s
- Avg followers
- ~200; celebrities up to 100M+
- Media/day
- 100M photos × 2 MB ≈ 200 TB/day
- Feed cache
- 500M users × ~500 ids × 16B ≈ 4 TB
APIs
- POST /posts
- createPost(userId, mediaId, caption) → postId
- GET /feed
- getFeed(userId, cursor, limit) → Post[]
- POST /follow
- follow(userId, targetId) → 200
- POST /media
- uploadMedia(userId, bytes) → mediaId + urls
Components
- API gateway: auth, rate limiting, routing.
- Post service + Post DB: sharded store of post metadata (author, caption, media ref, timestamp).
- Fanout workers: on new post, push the postId into followers' feed caches (for normal users).
- Feed cache (Redis): per-user list of recent post IDs — the precomputed timeline.
- Feed service: assembles a feed by merging cached IDs (fanout-on-write) with pulled celebrity posts (fanout-on-read), then hydrates.
- Media pipeline + CDN: resizes uploads to multiple resolutions and serves them from edge caches.
Request Flow
- 1User uploads a photo; media pipeline stores multi-resolution variants in blob storage and returns a mediaId.
- 2Post service writes post metadata to the sharded Post DB.
- 3Fanout workers append the postId to each follower's Redis feed list (skipping celebrity followers).
- 4On feed read, the Feed service reads the user's cached post IDs.
- 5It also pulls recent posts from the celebrities the user follows (fanout-on-read) and merges by timestamp.
- 6Post metadata is hydrated from cache/DB; the client loads media directly from the CDN.
Flow Diagram
Data / Storage Design
Post metadata lives in a sharded store keyed by postId (or userId), separate from media. The social graph (followers) is stored in its own service, sharded by userId, since follower lists are read on every fanout.
The feed cache is the crux: a Redis list per user holding the newest ~hundreds of post IDs. It is a materialized view, cheaply rebuildable from posts + graph, so it can be evicted for inactive users to save memory. Media bytes live in blob storage fronted by a CDN; only immutable URLs referencing a content hash flow through the app tier.
Data Model
Scaling Strategy
The hybrid fanout is the scaling lever. Fanout-on-write precomputes feeds so reads are cheap — great for the 99% of users with modest follower counts. But pushing a celebrity's post to 100M feeds is wasteful, so those are handled by fanout-on-read: the reader pulls the celebrity's recent posts at feed-build time and merges them in.
Feed caches, post DB, and the graph service all shard by user/post ID. The CDN offloads virtually all media bandwidth from origin. Inactive users' feed caches are lazily rebuilt to bound Redis memory.
Bottlenecks & Failure Scenarios
- Celebrity fanout storm: writing to tens of millions of caches — solved by fanout-on-read for high-follower accounts.
- Redis feed cache eviction/loss: rebuild feed lazily from posts + graph; degrade to fanout-on-read.
- Hot shard: a viral post's counters (likes) hammer one shard — use sharded counters / write-back aggregation.
- Thundering herd on cache miss: request coalescing and cache-aside with jittered TTLs.
- Media origin overload: absorbed by CDN edge caching; origin sees only cache fills.
Design Tradeoffs
- Precomputed feeds make the read path (the hot path) extremely fast.
- Hybrid fanout avoids the celebrity write-amplification trap.
- CDN + immutable media URLs slash bandwidth and latency.
- Feed cache storage is large and must be actively managed.
- Eventual consistency: a new post may take seconds to appear in all feeds.
- Merging two fanout strategies at read time adds code complexity.
Common Interview Questions
Fanout-on-write precomputes feeds and is ideal for normal users (cheap reads). Fanout-on-read pulls at query time and suits celebrities, where pushing to millions of caches would be prohibitive. Instagram uses a hybrid keyed on follower count.
Cap each user's list length, and only maintain caches for active users. Inactive users' feeds are evicted and rebuilt lazily on next login from posts + graph.
Photos are pre-resized into fixed resolutions, stored in blob storage with content-hash URLs, and served from a CDN so 99%+ of requests never reach origin.
- →Feed = precomputed Redis list of post IDs per user.
- →Fanout-on-write for normal users; fanout-on-read for celebrities.
- →Post metadata sharded and separate from media bytes.
- →CDN serves multi-resolution media via immutable URLs.
- →Feed cache is a rebuildable materialized view — evict freely.
Related
Everything on Archprimer is connected — follow the graph.