A System Designer guide
Caching, load balancing and queues, explained by breaking things
Three small experiments show what caching, load balancing and queues protect, what they cost and how each can fail.
These parts solve different problems. A load balancer shares work, a cache avoids repeated work and a queue postpones work. They often appear together, but adding all three without an explanation makes a failure harder to understand.
Experiment: spread the same work
Open Spreading load. Before touching the controls, predict what happens when several servers exist but clients still send requests to one address. Extra capacity only helps if requests can reach it.
A load balancer selects a healthy server for each request. Remove one server and ask whether the remaining pool has enough room. Spreading work does not create capacity, and health routing does not make a slow database faster. Watch the whole path rather than stopping at the front-end pool.
Compare an even rotation with choosing the least busy server when requests take different lengths of time. Then consider held-open connections: moving new requests is easier than moving an existing conversation. State which load the balancing policy actually sees.
Experiment: avoid repeated work
Open Remembering answers. Its database is nearly full and repeated answers may be at most 60 seconds old. Put a cache on the request path and predict how many reads still reach the database. The lesson’s check questions ask how hit rate changes that work; remember that writes still need their own path.
A time to live, or TTL, bounds how long an answer may remain in that cache without refresh. More memory does not make an old answer fresh. Long-lived copies help stable public data, while balances or private responses need a different policy.
Now use the URL shortener to test a cache stampede. When a copy empties, simultaneous misses can repeat the original work. Combining identical fetches limits that rush; spare storage capacity still matters for unrelated misses.
Experiment: spend time instead of dropping work
Open Waiting instead of breaking. Work enters a queue before workers send it to the destination. Predict whether the backlog grows by comparing arrival and completion rates. Only completion capacity above new arrivals is available to drain old work.
A queue can make a short burst survivable, but it cannot make an indefinitely overloaded destination keep up. Watch the oldest waiting job against its deadline. A smaller queue with very old work can be worse for users than a larger queue that drains quickly.
Try the Notification service next. Its providers have quotas. Increasing worker count beyond a quota adds contention or retries, not successful sends. Admission control tells callers when work cannot be accepted safely; backpressure slows producers when downstream work is already full.
Combine them by following one request
Say which step returns a reply. A cache hit may end before the application reaches storage. An accepted queued job may return before its external effect happens. A load balancer chooses a server, but that server still waits for its dependencies.
- For balancing, measure healthy capacity and the work left after one unit fails.
- For caching, measure hit rate, freshness and load while copies are cold.
- For queues, measure arrivals, drain rate, oldest age and the deadline.
- For all three, compare the monthly cost with the promise they protect.
Keep the workload fixed when comparing designs. Change one policy at a time and say what result would disprove your explanation. That habit is more useful than remembering where these boxes usually sit in a reference diagram.