Metrics for many customers

The brief

Build one metrics platform for 5,000 customers. A tenant can create millions of series or run a costly month-long query by mistake. Your design must preserve useful ingestion and query service for the other tenants, and show the offending customer a clear limit rather than an unexplained gap.

Think of a shared metrics service like Mimir.

Users, scale and requirements

Who uses it

5,000 tenant organizations (approx.)

Audience in plain numbers

25,000 users or devices

Normal peak

≈ 8,000 requests per second

Request mix

Telemetry batch: 90%; Report: 5%; Read: 5%

Speed for 99 of 100 requests

Report: ≈ 5,000 ms; Read: ≈ 1,000 ms

Successful requests

99.9%

Data safety

At most 1 minute of writes may be lost.

Monthly budget

≈ $19,000

Data at rest

≈ 1.2096 TB

Where users are

North America: 100%

Home region

North America

Waiting deadlines

Telemetry batch: 3,600 seconds; Report: 3,600 seconds; Read: 900 seconds

Minimum retained history

14 days on authoritative stores

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

• Each telemetry request batches 100 samples. Normal load is 10 million active series sampled every 15 seconds. Budget 1.5 B/sample for 14 days of compressed payload, before indexes, recent WAL and replica copies (approx.).
• Assume 100,000 active series per tenant at 2 KB of state per series, before sample buffers (approx.).
• The Time-series database element includes recent data and historical object blocks. The graph does not draw a fake store-to-store link.
• The shared API coordinates ingest and query requests. The sim does not model per-tenant weights, cardinality memory, new-series churn or cell assignment. The cardinality event affects the platform's own Metrics observer bill.
• Sources emit only their assigned request types. The source rates below reproduce the aggregate workload; protocol guarantees still require the separate design checkpoint.
• Scored data retention: at least 14 days on active authoritative stores. Copies used only for search or reports can be rebuilt. Local device history and rollup policies still require the separate protocol check.

Questions that shape the design

How many series and samples?

The normal fleet has about 10 million active series at a 15-second interval, roughly 667,000 samples/s. The peak plan is 720,000 samples/s. A telemetry request is a catalog batch of 100 samples (approx.).

What limits matter?

Enforce active-series, new-series and samples/s quotas plus label size on writes. Limit query concurrency, time range, scanned samples and wall time separately. Global requests/s is only a coarse admission guard.

How do customers share capacity?

Use weighted fair queues with bounded waiting, and shuffle-sharded ingest subsets inside independent cells. Cell-local data paths should isolate a failure. Shared identity and control-plane dependencies must be included in the blast-radius analysis.

What can be late?

Normal query visibility should be within a minute. Dashboards may cache stable ranges for a minute, but alert evaluations need a separate fresh path. Keep 14 days of full detail, then preserve semantics-aware aggregates under a separate policy.

The traffic day

Fairly steady traffic throughout the day. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

1. 08:00 · Metrics explosion

   A metric label creates many more distinct series, increasing the bill.

   Billed usage multiplier: 20. Duration: ≈ 21,600 seconds.

2. 10:00 · Noisy neighbour

   A competing workload reduces the capacity available to the live service.

   Available capacity multiplier: 0.5. Duration: ≈ 3,600 seconds.

3. 13:00 · Zone outage

   An availability zone loses power or network access.

   Duration: ≈ 3,600 seconds.

What a strong design has to handle

• Separate series count from sample count. Which customer can exhaust memory without sending more bytes?
• Enforce per-tenant active-series and query budgets with visible rejection.
• In a fixed-set quorum design, R = W = 2 of three replicas intersects, but reconciliation and the acknowledgement failure boundary still need a protocol. Mimir ingest storage uses Kafka durability rather than this ingester write quorum. Query visibility must account for consumer lag.
• A 100,000-series quota at an assumed 2 KB per active series is about 200 MB state per tenant before replication. Multiply only actual active series, not every tenant's unused quota reservation, unless the product guarantees reserved capacity.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

• How metrics systems work
• Many customers on one system
• Copies and quorums
• Consistent hashing and moving data
• SLOs and error budgets