A System Designer guide

Rate limiter walkthrough

Build fair admission for a public API: estimate counters, choose a sharing policy and explain what happens when the limiter cannot answer.

By System Designer · 7 November 2026

This walkthrough uses the Rate limiter for a public API scenario. Try an attempt first if you want to make your own choices before reading a possible approach.

Requirements and scale

Developers call your public API from scripts and apps. A few misbehaving or malicious clients can crowd out everyone else. Build limits that are fair, fast and cheap, and that survive a retry storm.

Who uses it: 2 million developers
Audience in plain numbers: 2,000,000 users or devices
Normal peak: ≈ 20,000 requests per second
Request mix: Read: 85%; Write: 15%
Speed for 99 of 100 requests: Read: ≈ 250 ms; Write: ≈ 250 ms
Successful requests: 99.9%
Data safety: At most 1 minute of writes may be lost.
Monthly budget: ≈ $89,000
Data at rest: ≈ 1 TB
Where users are: North America: 60%; Europe: 40%
Home region: North America
Waiting deadlines: Read: 900 seconds; Write: 3,600 seconds

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

Estimate memory for 2 million counters at about 100 bytes per key, including overhead.
Typical limit: 5,000 requests an hour per key, bursts allowed (approx.).

Estimates to work through

Writes per second at peak: ≈ 3,000 writes/s. 20,000 × 0.15
Counter memory for 2 million keys at 100 bytes each (approx.): ≈ 200 MB. 2 million keys × 100 bytes (approx.)
Limit check cost: ≈ 0.2 ms. One in-memory counter update, about the speed of a cache hit (approx.)

Decide what the limit means

A limit needs an identity, a time window and a policy for bursts. A developer key is a better starting point than an IP address when many legitimate users share a network. Decide whether the allowance is global or local to a region. That choice determines how counters must be shared.

A token bucket is one way to allow a short burst without permitting an unlimited sustained rate. Tokens accumulate up to a ceiling; a request spends one. A fixed window is easier to explain but can admit a burst on both sides of a boundary. Choose according to the promise, not the name of the algorithm.

A design that holds

Put admission before expensive application work. Filter obvious attack traffic, identify the caller, check the allowance in fast memory and pass admitted work to the service. Rejected callers need a clear response and a sensible retry time so the limiter does not create its own retry storm.

The read, decision and update of a shared counter need one atomic operation. Separate reads and writes let simultaneous requests spend the same allowance. Partition counters by caller, then examine whether one huge caller can still overload a single key.

The trade-offs

Shared counters give a more exact combined allowance but add a network dependency. Independent local counters remove that hop but multiply the total permitted traffic unless the quota is divided. Divided quotas can leave one region rejecting callers while another has spare allowance.

Choose an outage policy explicitly. Failing open keeps ordinary clients moving when the limiter fails, while exposing the service to abuse. Failing closed protects capacity but rejects legitimate work too. A limited local fallback can reduce the damage, but its allowance and reconciliation still need explaining.

What breaks it

A bot may have no useful key, a hot caller may overwhelm one counter, and clients may retry faster than useful work completes. Test those separately. Buying more application capacity does not make a per-customer policy fair or a counter update atomic.

Use Saying "slow down" politely to practise the admission rule and Many customers on one system to reason about fairness. In the Rate limiter for a public API, predict which work the limiter removes and explain the uptime cost of rejecting it.

The day that tests the design

Traffic concentrated in working hours. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

10:00 · Bot scraping
Bots try stolen passwords and scrape pages, adding unwanted traffic.
Extra attack traffic is 1 times the configured peak. Duration: ≈ 7,200 seconds.
15:30 · Retry storm
A brief failure causes clients to retry, adding load while the service recovers.
Failed calls: 90%. Duration: ≈ 30 seconds. Duration: ≈ 1,200 seconds.

Play this problem. Keep one baseline, predict a change and compare the next full day. The in-game editorial has calibrated reference designs after you finish a traffic day.