Design Dropbox

You're asked to design a file sync service like Dropbox or Google Drive. A user drops a file into a folder on their laptop, and it should appear on their phone and their desktop shortly afterward. It should handle files from a few kilobytes to several gigabytes, work when a device has been offline for a week, keep version history, and support sharing a folder with other people.

Uploading a file to cloud storage is straightforward. What makes this hard is that the same folder exists on several devices at once, all of them editing independently, and the system has to converge them without losing anyone's work.

Clarifying the requirements

• How large can files be? Gigabyte files force chunked, resumable uploads. This is the assumption worth confirming first.
• Do we need version history? Keeping every version changes the storage math and enables restore. Most versions of this question include it.
• Is sharing in scope? Sharing a folder with another user introduces permissions and multi-writer conflicts, and roughly doubles the scope.
• What happens on a conflict? Two devices edit the same file offline. Ask whether the product wants last-write-wins, both copies kept, or a merge.
• How fast should sync be? Seconds when online is the usual expectation, which rules out polling every few minutes.

Assume: files up to several GB, version history, folder sharing, and conflicts resolved by keeping both copies.

Back-of-envelope numbers

• 100M users, 10% active daily, ~10 file changes each → $10\text{M} \times 10 = 100\text{M changes/day} \approx 1{,}200/\text{sec}$
• Storage: $100\text{M users} \times 20\text{ GB avg} \approx 2\text{ EB}$ before deduplication
• Metadata: $100\text{M users} \times 5{,}000 \text{ files} \times 500\text{ B} \approx 250\text{ TB}$

The metadata figure is the one worth dwelling on. A quarter of a petabyte of small records under constant write pressure is a partitioning problem you have to solve yourself, whereas the exabytes of file content can be delegated to object storage.

High-level architecture

Deep dive 1: chunking and deduplication

Files are split into fixed-size chunks, typically around 4 MB, and each chunk is hashed. The file's metadata record is then an ordered list of chunk hashes rather than the content itself.

This one decision produces several properties at once:

• Resumable uploads. A dropped connection costs you the current chunk, not the whole file.
• Delta sync. Editing one paragraph of a large document changes one chunk, so the client uploads a few megabytes rather than the whole file.
• Deduplication. Chunks are addressed by content hash, so the same chunk uploaded by a thousand users is stored once. For commonly shared files this is a very large saving.
• Cheap versioning. A new version is a new list of chunk hashes, most of which already exist. Version history costs the size of what changed rather than a full copy.

Before uploading, the client asks the service which chunk hashes it already holds and sends only the missing ones. On a re-upload of a mostly unchanged file, that exchange can eliminate nearly all of the transfer.

Deep dive 2: detecting changes without polling

Every device needs to learn about changes made elsewhere, and having millions of clients poll a server every few seconds is both slow to react and expensive to serve.

Instead, each device holds a long-lived connection to a notification service, either a WebSocket or a long-poll, which tells it only that something changed in a namespace it cares about. The notification carries no file content, which keeps that service small and cheap. On receiving one, the device calls the metadata service to ask what changed since the version it last saw, and pulls the chunks it's missing.

The mechanism that makes this reliable is a per-namespace version cursor: a monotonically increasing number bumped on every change. Each device stores the last cursor it processed, and sync means "give me everything after cursor N." A device that's been offline for a week doesn't need any special handling. It just asks with an old cursor and receives more changes than usual.

Deep dive 3: conflicts

Two devices edit the same file while both are offline. Both come back online and commit. Neither change is wrong, and the system has to decide what happens.

Last-write-wins by timestamp is simple and silently destroys one person's work. It also relies on device clocks, which are not trustworthy.

Keeping both versions is what real file sync products do. The second writer's file is preserved alongside the first under a modified name, the "conflicted copy" you may have seen in a Dropbox folder, and the user decides which to keep. The system has no way to know which edit was meant to win, so escalating to the person who made the edits is the only correct behavior.

Detection uses versions rather than timestamps. Each file record carries a version, and a commit includes the version the client believed it was editing. If that doesn't match the current version, the server knows the client edited a stale copy and creates a conflicted copy instead of overwriting. This is optimistic concurrency, the same mechanism described in transactional workflows.

Text documents are a special case where genuine merging is possible, which is the domain of real-time and collaborative systems. For arbitrary binary files, keeping both is the only correct answer.

Deep dive 4: partitioning the metadata

The metadata database is the component that has to scale, and the partition key follows from the access pattern. Nearly every query is "what's in this user's namespace" or "what changed in this shared folder," so partitioning by user or by workspace keeps those queries inside a single partition.

Sharing complicates this. A folder shared between users in different partitions means a change has to be visible to both. The usual approach is to make the shared folder its own namespace with its own cursor, and give each member a reference to it, rather than duplicating its contents into every member's namespace.

Two more details worth raising:

• Hot namespaces. A folder shared with 500 people generates far more traffic than a personal one, which is a hot partition and may deserve dedicated capacity.
• Chunk garbage collection. Deleting a file cannot immediately delete its chunks, since other files and other versions may reference them. Reference counting with delayed cleanup handles this, and mentioning that deletion is asynchronous shows you've thought past the happy path.

Common pitfalls

• Uploading whole files. No resume, no delta sync, and no deduplication.
• Routing bytes through the application tier. Clients should read and write object storage directly with presigned URLs.
• Polling for changes. Slow to react and expensive at scale compared to a notification channel plus a cursor.
• Last-write-wins on conflicts. Silently loses work, and depends on unreliable device clocks.
• Immediate chunk deletion. Breaks other files and older versions that share those chunks.

Leveling the answer

Related lessons