Hard · Developer platforms
Logs platform
Build a shared log service for 2,000 teams. Collectors send compressed chunks, and on-call engineers find logs by tenant, service and time before filtering their words. A wide search or a runaway label must not stop other teams from investigating an outage. Practise this hard system design problem for free with a Google or GitHub account.
The brief
Build a shared log service for 2,000 teams. Collectors send compressed chunks, and on-call engineers find logs by tenant, service and time before filtering their words. A wide search or a runaway label must not stop other teams from investigating an outage.
Think of a label-indexed log service like Loki.
Users, scale and requirements
- Who uses it
- 2,000 tenant teams (approx.)
- Audience in plain numbers
- 20,000 users or devices
- Normal peak
- ≈ 100 requests per second
- Request mix
- Batch task: 1%; Read: 99%
- Speed for 99 of 100 requests
- Read: ≈ 1,000 ms
- Successful requests
- 99.9%
- Data safety
- No acknowledged write may be lost.
- Monthly budget
- ≈ $8,800
- Data at rest
- ≈ 259.2 TB
- Where users are
- North America: 100%
- Home region
- North America
- Waiting deadlines
- Batch task: 3,600 seconds; Read: 900 seconds
- Minimum retained history
- 30 days on authoritative stores
≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.
- Each compressed chunk is about 100 MB. Plan 30 days of payload before metadata and audit exceptions. A broad scan reads one full day of chunks at 2 GB/s (approx.).
- The score measures durable chunk acceptance and small reads. It does not model label index construction, text scan work, tenant fairness or redaction.
- A catalog batch represents a 100 MB object upload, not one message sent through a standard queue. Object chunks are immutable. Stable chunk ids and sequence ranges make retries reconcilable; index publication follows durable upload.
- Sources emit only their assigned request types. The source rates below reproduce the aggregate workload; protocol guarantees still require the separate design checkpoint.
- Scored data retention: at least 30 days on active authoritative stores. Copies used only for search or reports can be rebuilt. Local device history and rollup policies still require the separate protocol check.
Questions that shape the design
Do we need arbitrary word search with no service or time filter?
It is allowed as a bounded asynchronous scan. Normal interactive queries specify tenant, service and a narrow time range. A label index is enough for this path; a full-text index is an optional paid tier.
What counts as an accepted chunk?
The API verifies a stable tenant-scoped chunk id and checksum, durably uploads the chunk, then acknowledges it. Search visibility may lag by up to a minute while index metadata is published. Collectors retain unacknowledged chunks and retry the same identity with backoff.
What is retained?
Keep ordinary compressed logs for 30 days and selected audit streams for a separate agreed retention policy. Redact secrets at collection. Never use request ids or user ids as stream labels.
How is the traffic represented?
The batch type represents a 100 MB compressed chunk, as in the catalog. Read represents a small chunk or index lookup, not an arbitrary full-text scan. Query fan-out, index cardinality and scanned bytes need separate sizing.
The traffic day
Fairly steady traffic throughout the day. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.
09:00 · Noisy neighbour
A competing workload reduces the capacity available to the live service.
Available capacity multiplier: 0.5. Duration: ≈ 3,600 seconds.
12:00 · Zone outage
An availability zone loses power or network access.
Duration: ≈ 3,600 seconds.
18:00 · Retry storm
A brief failure causes clients to retry, adding load while the service recovers.
Failed calls: 90%. Duration: ≈ 30 seconds. Duration: ≈ 1,200 seconds.
What a strong design has to handle
- Choose the index from the normal query shape. Which labels keep stream count bounded?
- Bound tenant scans and label growth while keeping durable ingestion usable.
- A label index avoids continuous full-text indexing cost, but broad word searches scan more chunk bytes. Bound them and show pending results.
- Never publish an index pointer before its immutable chunk is durable. Orphan chunks can be reconciled later; dangling pointers create missing results. Retries reuse chunk ids and sequence ranges.
Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.
Concepts to practise
Related problems
- Hard
Tracing platform
Store distributed traces for 500 teams and make an exemplar link open the useful trace during an incident. Decide what to sample before storage, handle late spans, and keep query traffic from overwhelming ingestion. Keeping all errors is a policy goal with a failure boundary, not a magic property of sampling.
- Hard
Metrics platform
Thousands of services send a stream of measurements: CPU, errors, latency. You store them, draw the dashboards and fire the alerts. The platform watching everyone else must never go down first, and one customer creating millions of new series can ruin the bill.
- Hard
Metrics for many customers
Build one metrics platform for 5,000 customers. A tenant can create millions of series or run a costly month-long query by mistake. Your design must preserve useful ingestion and query service for the other tenants, and show the offending customer a clear limit rather than an unexplained gap.
- Hard
Object storage service
Customers store billions of files, from tiny pictures to huge videos, and fetch them by name. The service must never lose a file, scale without a limit and stay cheap. Most files are written once and rarely read again.