Dropbox / File Sync
A file sync and storage service using content-defined chunking, block-level deduplication, a metadata service, and sync with conflict resolution across devices.
By Ratish Jain · Updated
Problem
Design Dropbox: keep a user's files synchronized across all their devices, storing them durably in the cloud, while minimizing bandwidth and storage.
The key insight is to separate metadata from file content. Files are split into chunks, and only chunks that changed are uploaded — so editing one page of a document syncs a few KB, not the whole file. Block-level deduplication avoids storing identical chunks twice. The tricky part is sync and conflict resolution when the same file is edited on two devices.
Functional Requirements
- Upload, download, and delete files; preserve folder structure.
- Automatically sync changes across all a user's devices.
- Upload only changed chunks (delta sync) to save bandwidth.
- Deduplicate identical content across files/users.
- Detect and resolve conflicting concurrent edits.
- Offline edits sync when the device reconnects.
Non-Functional Requirements
- Durable storage — never lose a committed file (11 nines style).
- Bandwidth-efficient via chunking + dedup.
- Sync latency of a few seconds for online devices.
- Strong consistency of metadata; content is immutable once written.
- Scalable to hundreds of millions of users and exabytes of data.
Capacity Estimation
- Registered users
- ~700M
- Avg storage/user
- ~10 GB → ~7 EB total
- Chunk size
- ~4 MB fixed/content-defined blocks
- Files/day changed
- ~1B → dedup cuts uploads sharply
- Metadata rows
- billions of file/block mappings
- Dedup savings
- large — many identical blocks across users
APIs
- POST /commit
- commit(path, blockList, baseVersion) → version
- POST /blocks
- putBlock(hash, bytes) → 200 (skip if exists)
- GET /blocks/:hash
- getBlock(hash) → bytes
- GET /changes
- getChanges(cursor) → ChangedFile[]
Components
- Client sync agent: watches the local folder, chunks files, computes hashes, and reconciles with the server.
- Metadata service + DB: authoritative file/folder tree, versions, and file→block mappings (sharded, strongly consistent).
- Block service: content-addressed store — accepts a block only if its hash isn't already present.
- Block store (S3-like): durable, replicated object storage for immutable chunks.
- Notification service: pushes 'something changed' pokes to a user's other online devices.
- Dedup index: hash → block location, enabling cross-file/user deduplication.
Request Flow
- 1Client detects a changed file, splits it into chunks, and hashes each chunk.
- 2It asks the block service which hashes are missing; only new blocks are uploaded to block storage.
- 3The client commits the new file version (path + ordered block list) to the metadata service with its base version.
- 4Metadata service validates the base version (optimistic concurrency) and persists the new version.
- 5It emits a change event; the notification service pokes the user's other online devices.
- 6Those devices pull the changed metadata, fetch any missing blocks, and reassemble the file locally.
Flow Diagram
Data / Storage Design
Content is stored as immutable, content-addressed blocks: the block's key is the hash of its bytes, so identical content is naturally deduplicated and blocks can be cached and CDN'd safely. Blocks live in replicated object storage for durability.
Metadata is the source of truth and the consistency boundary: a file version is just a path plus an ordered list of block hashes, stored in a sharded, transactional DB (sharded by user/namespace). Because content is separated from metadata, delta sync is trivial — a new version references mostly the same blocks plus a few new ones. A journal / changes cursor per namespace lets devices pull only what changed since they last synced.
Data Model
Scaling Strategy
Block storage scales as content-addressed objects — effectively infinite via object storage, with global dedup shrinking real footprint. Blocks are immutable, so they're trivially cacheable and CDN-friendly for download.
Metadata is sharded by user/namespace, keeping a user's file tree and change journal co-located for fast, consistent sync operations. The notification service uses long-lived connections to fan out change pokes; devices then pull deltas (metadata is small), so the heavy block transfer only moves genuinely new content.
Bottlenecks & Failure Scenarios
- Concurrent edits (conflict): optimistic concurrency on baseVersion; a stale commit is rejected and the client creates a 'conflicted copy'.
- Metadata hotspot: a shared folder with many editors — shard by namespace and batch change events.
- Partial upload / crash mid-sync: blocks are content-addressed and idempotent; commit is atomic, so an incomplete upload never corrupts a version.
- Hash collision risk: use a strong hash; astronomically unlikely but handled by verification.
- Notification service down: clients fall back to periodic polling of the changes cursor.
Design Tradeoffs
- Chunking + dedup drastically cut bandwidth and storage.
- Immutable content-addressed blocks are cacheable and simplify consistency.
- Separating metadata from content makes delta sync cheap and clean.
- Conflict resolution can't fully avoid user-facing 'conflicted copy' files.
- Metadata is the consistency bottleneck and must be carefully sharded.
- Dedup index and block GC (deleting truly-unreferenced blocks) add complexity.
Common Interview Questions
The client splits files into chunks and hashes each. On a change, only chunks whose hashes changed are uploaded; the new file version references the unchanged blocks plus the few new ones. This is delta sync via content-addressed blocks.
Metadata commits carry the base version. The server accepts the first commit; a second commit against a now-stale base is rejected, and that client saves its version as a 'conflicted copy' so no edit is silently lost.
Blocks are large, immutable, and best in cheap object storage; metadata is small, mutable, and needs strong consistency. Separating them lets each scale and be consistency-managed independently, and makes dedup and delta sync natural.
- →Split files into chunks; upload only new blocks (delta sync).
- →Content-addressed blocks (hash keys) give free dedup + caching.
- →Metadata service is the source of truth, sharded by namespace.
- →Optimistic concurrency handles conflicts via 'conflicted copy'.
- →Notify service pokes devices; they pull small metadata deltas.
Related
Everything on Archprimer is connected — follow the graph.