Archprimer
ArchitectureAdvanced 6 min read

YouTube

A video platform covering the async upload/transcode pipeline, adaptive-bitrate streaming, global CDN delivery, and metadata storage for search and playback.

CDNTranscodingABRStreamingQueue

By Ratish Jain · Updated

Diagram with 7 components: Creator, Upload Svc, Raw Store, Job Queue, Transcoders, CDN, Viewer. Connections: Creator → Upload Svc (1 · upload); Upload Svc → Raw Store (2 · store); Upload Svc → Job Queue (3 · enqueue); Job Queue → Transcoders (4 · transcode); Transcoders → CDN (publish); Viewer → CDN (5 · stream).
ClientServiceStorageQueueCDN
Uploads land in blob storage, get transcoded asynchronously into ABR renditions, then are served worldwide via CDN.

Problem

Design YouTube: creators upload videos of arbitrary size and format; viewers stream them smoothly on any device and network.

The defining challenges are the transcode pipeline (turning one uploaded file into many resolution/bitrate renditions asynchronously) and delivery at planetary scale via CDN with adaptive bitrate (ABR) so playback adjusts to changing bandwidth. Metadata (titles, views, search) is a comparatively conventional read-heavy system.

Functional Requirements

  • Upload videos of arbitrary size/codec; resumable on failure.
  • Transcode into multiple resolutions/bitrates (ABR ladder).
  • Stream with adaptive bitrate that adjusts to bandwidth.
  • Browse, search, and view metadata (title, views, thumbnails).
  • Track view counts and basic engagement.

Non-Functional Requirements

  • Playback start (join time) under ~2 s; minimal rebuffering.
  • Durable storage — uploaded originals never lost.
  • Massively read-heavy delivery; global low-latency reach.
  • Upload availability decoupled from transcode completion.
  • Cost-efficient storage and egress at exabyte scale.

Capacity Estimation

Uploads/min
~500 hours of video/min
Views/day
~5B → ~60K playback starts/s
Avg upload size
~1 GB raw before transcode
Renditions per video
~6 (144p → 4K) × codecs
Storage growth
~1 PB+/day after renditions
Peak egress
~100+ Tbps served from CDN edges

APIs

POST /uploads
initUpload(userId, meta) → uploadId, urls
PUT /uploads/:id
putChunk(uploadId, index, bytes) → 200
GET /videos/:id
getMetadata(videoId) → {manifestUrl, meta}
GET /manifest
getManifest(videoId) → HLS/DASH renditions

Components

  • Upload service: accepts chunked, resumable uploads and writes originals to raw blob storage.
  • Raw store: durable object storage for uploaded originals.
  • Job queue (Kafka): enqueues transcode jobs, decoupling upload from processing.
  • Transcoder fleet: splits video into segments and encodes the full ABR ladder in parallel.
  • Processed store + CDN: segmented renditions + HLS/DASH manifests, distributed to edge caches.
  • Metadata service: video info, search index, and view counters in a sharded DB.
  • Player: fetches the manifest and switches renditions based on measured bandwidth.

Request Flow

  1. 1Creator uploads in chunks; the upload service assembles and stores the original in raw blob storage.
  2. 2A transcode job is enqueued to Kafka; the video is marked 'processing'.
  3. 3Transcoders pick up the job, split it into segments, and encode each into the ABR ladder in parallel.
  4. 4Segmented renditions + HLS/DASH manifest are written to the processed store and pushed to the CDN; metadata flips to 'ready'.
  5. 5A viewer requests the video; the player fetches the manifest and begins streaming segments from the nearest CDN edge.
  6. 6The player continuously measures throughput and switches up/down the bitrate ladder to avoid rebuffering.

Flow Diagram

Diagram with 5 components: Uploader, Upload API, Raw Store, Transcode Q, Transcoder. Connections: Uploader → Upload API (1 · upload); Upload API → Raw Store (2 · store raw); Upload API → Transcode Q (3 · enqueue); Transcode Q → Transcoder (4 · transcode); Transcoder → Raw Store (read).
ClientGatewayStorageQueueService
Upload lands in raw store; a queue drives transcoding.

Data / Storage Design

Two storage tiers: raw originals (write-once, cold, cheap) and processed renditions (segmented .ts/fMP4 files + manifests, hot, CDN-fronted). Video is chunked into a few-second segments so transcoding parallelizes and ABR can switch bitrate at segment boundaries.

Metadata (title, description, uploader, thumbnails, tags) lives in a sharded relational or wide-column store keyed by videoId, with a separate search index. View counts are high-write and are handled with sharded counters / async aggregation rather than a single hot row.

Data Model

Diagram with 5 components: Transcoder, Raw Store, Renditions, Metadata DB, CDN. Connections: Transcoder → Raw Store (1 · read raw); Transcoder → Renditions (2 · write); Transcoder → Metadata DB (3 · meta); Renditions → CDN (4 · publish).
ServiceStorageDatabaseCDN
Raw + rendition media stores; metadata DB; publish to CDN.

Scaling Strategy

Transcoding scales horizontally and embarrassingly parallel: because video is segmented, a fleet of workers encodes different segments and renditions simultaneously, so a long video isn't a long serial job. The queue absorbs upload bursts and smooths transcoder load.

Delivery scales through the CDN — the vast majority of bytes are served from edge caches near viewers, so origin and processed store see mostly cache fills. Popular videos are pre-warmed to edges. Metadata reads are cached; counters are sharded to avoid hotspots.

Bottlenecks & Failure Scenarios

  • Transcode backlog: viral upload spikes lengthen the queue — autoscale worker fleet, prioritize by expected popularity.
  • Transcoder crash mid-job: jobs are idempotent per segment; failed segments are retried, not the whole video.
  • CDN cache miss storm on a brand-new viral video — pre-warm edges and use origin shielding.
  • Hot view counter: single-row contention — shard counters and aggregate asynchronously.
  • Storage cost blowup: dedupe identical renditions, tier cold originals to cheaper storage.

Design Tradeoffs

Pros
  • Async transcode decouples upload success from heavy processing.
  • Segmentation enables both parallel encoding and seamless ABR switching.
  • CDN + edge pre-warming makes global delivery cheap and fast.
Cons
  • Video isn't watchable until transcode completes (processing delay).
  • Storing many renditions multiplies storage cost.
  • ABR and multi-codec support add player and pipeline complexity.

Common Interview Questions

Why transcode asynchronously instead of on upload?

Transcoding a 1 GB file into six renditions is CPU-heavy and slow. Doing it inline would make uploads time out and couple availability to processing. A queue lets uploads finish instantly and workers process at their own pace, with autoscaling for bursts.

How does adaptive bitrate streaming work?

Each video is encoded into multiple bitrate renditions and split into short segments described by an HLS/DASH manifest. The player measures throughput and requests the next segment from the highest bitrate that won't rebuffer, switching at segment boundaries.

How do you serve billions of views without melting origin?

A CDN caches segments at edges near viewers; origin only serves cache fills. Popular content is pre-warmed and origin shielding coalesces misses so origin sees a tiny fraction of traffic.

60-Second Revision
  • →Upload → raw store → Kafka job → transcode ladder → CDN.
  • →Async transcoding decouples upload from heavy processing.
  • →Segmentation = parallel encoding + ABR bitrate switching.
  • →CDN edges serve nearly all bytes; pre-warm viral videos.
  • →Metadata sharded by videoId; view counts sharded to avoid hotspots.

Related

Everything on Archprimer is connected — follow the graph.