Skip to content
System Designer

Medium · Productivity

File sync

A folder on your laptop mirrors to the cloud and to your other devices. Small changes must arrive quickly, big files must upload reliably, and a shared folder with thousands of people must not melt the metadata service. Practise this medium system design problem for free with a Google or GitHub account.

The brief

A folder on your laptop mirrors to the cloud and to your other devices. Small changes must arrive quickly, big files must upload reliably, and a shared folder with thousands of people must not melt the metadata service.

Think of a file sync app like Dropbox.

Users, scale and requirements

Who uses it
3 million people a month
Audience in plain numbers
3,000,000 people per month
Normal peak
≈ 4,000 requests per second
Request mix
Sync change: 66%; Read: 25%; Upload: 6%; Write: 3%
Speed for 99 of 100 requests
Sync change: ≈ 300 ms; Read: ≈ 300 ms; Upload: ≈ 3,000 ms; Write: ≈ 300 ms
Successful requests
99.5%
Data safety
No acknowledged write may be lost.
Monthly budget
≈ $54,000
Data at rest
≈ 900 TB
Where users are
North America: 50%; Europe: 50%
Home region
North America
Waiting deadlines
Sync change: 1,800 seconds; Read: 1,800 seconds; Upload: 1,800 seconds; Write: 1,800 seconds
Fresh reads
People must see their own changes. At most 0.1% of reads may be stale in the model.
Personal reads
95% of reads are private; a shared public copy cannot answer them.

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

  • Files are cut into blocks of about 4 MB, and identical blocks are stored once (approx.).
  • Shared folders are the hot spots.

Questions that shape the design

What are the two kinds of data?

File contents (large, in blocks) and file lists with names and versions (small, but constantly changing). They are stored differently.

What happens when two people change the same file?

Keep both versions and show a "conflicted copy". Never silently lose someone's work.

How fast must a change show up on another device?

Within a few seconds for small files. While a part of the service is being repaired, changes may wait in line up to half an hour; later than that counts as failed.

The traffic day

Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

  1. 09:00 · Synchronized rush

    Many callers send requests together instead of spreading them over time.

    Traffic for Sync change, Read reaches 6 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.

  2. 13:00 · Hot key

    One item gets a huge share of traffic: a celebrity profile, one product, one merchant.

    Affected share: 30%. Duration: ≈ 3,600 seconds.

  3. 15:00 · Disk full

    Data or logs fill the disk. Writes stop until someone frees space.

    Lasts until detected and repaired.

What a strong design has to handle

  • File blocks and file lists are different problems. Do they belong in the same database?
  • Resume chunk uploads and preserve concurrent file versions across devices.
  • Storing identical blocks once saves a lot of space but means a block is only deleted when nobody uses it.
  • Sharding by user spreads load. A shared folder with thousands of people becomes a hot spot that needs its own cache.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

Related problems