Hard · Productivity
Collaborative documents
Many people edit the same document at the same moment and everyone must see a consistent result in a fraction of a second. Edits must never be lost, even if a server crashes in the middle of typing or two people type in the same place. Practise this hard system design problem for free with a Google or GitHub account.
The brief
Many people edit the same document at the same moment and everyone must see a consistent result in a fraction of a second. Edits must never be lost, even if a server crashes in the middle of typing or two people type in the same place.
Think of shared docs like Google Docs.
Users, scale and requirements
- Who uses it
- 15 million people a month
- Audience in plain numbers
- 15,000,000 people per month
- Normal peak
- ≈ 40,000 requests per second
- Request mix
- Message: 60%; Sync change: 20%; Read: 15%; Write: 5%
- Speed for 99 of 100 requests
- Message: ≈ 250 ms; Sync change: ≈ 300 ms; Read: ≈ 300 ms; Write: ≈ 300 ms
- Successful requests
- 99.95%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $83,000
- Data at rest
- ≈ 60 TB
- Where users are
- North America: 50%; Europe: 50%
- Home region
- North America
- Waiting deadlines
- Message: 900 seconds; Sync change: 900 seconds; Read: 900 seconds; Write: 3,600 seconds
- Fresh reads
- People must see their own changes. At most 0.1% of reads may be stale in the model.
- Personal reads
- 90% of reads are private; a shared public copy cannot answer them.
- Required proofs
- Managed encryption keys
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- 250,000 people editing at once, about a tenth typing in any one second (approx.).
- One gateway node holds about 100,000 open connections (approx.).
Questions that shape the design
What if two people edit the same sentence together?
Both edits survive and everyone ends up with the same text. The order of edits is decided in one place per document.
How fast must an edit reach others?
Under about 250 ms, counting the trip across an ocean. More than that and cursors feel jumpy.
What if someone loses their connection?
They keep typing offline and their edits merge when they return.
Can we lose an edit?
Never. Typed text is saved before it is confirmed.
The traffic day
Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
09:00 · Synchronized rush
Many callers send requests together instead of spreading them over time.
Traffic for Read, Sync change, Message reaches 5 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.
11:00 · Zone outage
An availability zone loses power or network access.
Duration: ≈ 3,600 seconds.
14:30 · Network partition
Your regions cannot talk to each other during the partition.
Duration: ≈ 300 seconds.
What a strong design has to handle
- Two people editing the same document must agree on the order. Where does that order get decided?
- Converge concurrent edits and recover durable operations across reconnects.
- One owner per document gives a clear order but is a single point to recover. The coordination service picks a new owner within seconds.
- Writing to the log before confirming adds a few milliseconds but means no typed text is ever lost.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Medium
One-to-one chat
People send short messages to each other from their phones. A message must arrive once, in order, even when the phone is on a bad connection and sends it again. If the other person is offline, the message waits and a notification goes out.
- Medium
File sync
A folder on your laptop mirrors to the cloud and to your other devices. Small changes must arrive quickly, big files must upload reliably, and a shared folder with thousands of people must not melt the metadata service.
- Hard
Notes that sync across your devices
Build a notebook that works on a phone and laptop, including a month offline. Edits should reach connected foreground devices in under a second, concurrent work must not disappear, and the server should not need plaintext notes. Push wakes devices; durable sync must work when push does not.
- Hard
Scheduled job service
Customers register jobs: "call my web address every night at 02:00" or "every hour on the hour". Aim to fire within 5 seconds; work still waiting after 60 seconds counts as failed. Retry safely after a crash. One external effect requires the customer endpoint to honor a stable job occurrence key. Half the world picks the top of the hour, so a huge crowd of jobs comes due in the same second.