Firmware rollout

The brief

A new firmware goes out to a million smart devices at 03:00 when they are idle. Every device asks "is there an update?", then downloads the same file in pieces, then reports back. If they all wake together, they can flatten your service, and a device that fails to update must not be left broken.

Think of an over-the-air update for a million smart devices.

Users, scale and requirements

Who uses it

1 million devices (approx.)

Audience in plain numbers

1,000,000 users or devices

Normal peak

≈ 600 requests per second

Request mix

Read: 55%; Telemetry batch: 30%; Analytics event: 10%; Write: 5%

Speed for 99 of 100 requests

Read: ≈ 800 ms; Write: ≈ 800 ms

Successful requests

99.5%

Data safety

At most 1 minute of writes may be lost.

Monthly budget

≈ $4,900

Data at rest

≈ 2 TB

Where users are

North America: 50%; Europe: 30%; Asia Pacific: 20%

Home region

North America

Waiting deadlines

Read: 900 seconds; Telemetry batch: 3,600 seconds; Analytics event: 3,600 seconds; Write: 3,600 seconds

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

• The firmware is about 40 MB, cut into 1 MB pieces (approx.).
• Devices wake together in the first minutes after 03:00 (approx.).
• One app server handles about 1,000 requests a second (approx.).
• All numbers are rough approximations.

Questions that shape the design

Do all devices need the update at once?

No. Within a few hours is fine, so the rush can be spread out.

Are the files the same for everyone?

Yes, per model. The same pieces are asked for by hundreds of thousands of devices.

What if a device keeps failing?

It keeps its old firmware and tries again later. The service must tell us how many devices updated.

The traffic day

A quiet baseline followed by the scheduled rush. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

1. 03:05 · Synchronized rush

   Many callers send requests together instead of spreading them over time.

   Traffic for Sign-in, Message, Read, Sync change reaches 3 times the scheduled load; rises over 10 seconds, holds for 60 seconds and eases over 300 seconds. Other request types keep their scheduled load.

2. 14:00 · Network partition

   Your regions cannot talk to each other during the partition.

   Duration: ≈ 300 seconds.

What a strong design has to handle

• A million devices ask for the same few files. What should answer them, and what should never see them?
• Stage signed firmware and stop safely when devices fail the rollout.
• Rolling out to a few percent first catches a bad firmware before it reaches everyone, at the price of a slower rollout.
• Random delays on each device spread the rush for free, and cost nothing on the server side.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

• Copies near users
• Saying "slow down" politely