Skip to content
System Designer

Hard · Finance and payments

Real-time fraud detection

Every card payment asks your system "is this fraud?" and wants the answer within about 150 milliseconds. Each check looks up the cardholder's recent habits and runs a model. The holiday shopping rush doubles the volume, and a bad model update can start declining honest customers. Practise this hard system design problem for free with a Google or GitHub account.

The brief

Every card payment asks your system "is this fraud?" and wants the answer within about 150 milliseconds. Each check looks up the cardholder's recent habits and runs a model. The holiday shopping rush doubles the volume, and a bad model update can start declining honest customers.

Think of card fraud checks at a bank.

Users, scale and requirements

Who uses it
20 million cardholders
Audience in plain numbers
20,000,000 users or devices
Normal peak
≈ 5,000 requests per second
Request mix
Payment: 80%; Analytics event: 10%; Read: 5%; Report: 5%
Speed for 99 of 100 requests
Payment: ≈ 150 ms; Read: ≈ 300 ms; Report: ≈ 10,000 ms
Successful requests
99.95%
Data safety
At most 1 minute of writes may be lost.
Monthly budget
≈ $22,000
Data at rest
≈ 10 TB
Where users are
North America: 100%
Home region
North America
Waiting deadlines
Payment: 3,600 seconds; Analytics event: 3,600 seconds; Read: 900 seconds; Report: 3,600 seconds
Required proofs
Audit trail; Managed encryption keys

≈ marks an approximation, just as in the game. Waiting deadlines measure the age of queued work and are separate from interactive response time.

  • A "payment" here is one authorization check.
  • Analysts review flagged cases in a console.

Questions that shape the design

How long do we have for the answer?

About 150 ms in total. The card network gives up soon after.

What if we are too slow?

The payment fails or is approved without a check, both of which cost money. Staying fast matters more than being perfect.

Where do the cardholder's habits come from?

A feature store of recent activity, updated by a stream of events as payments happen.

Can a new model go wrong?

Yes. Releases need a way back in seconds, because declined honest customers call the bank immediately.

The traffic day

Consumer traffic, busiest in the evening. The clock covers 00:00 to 24:00 in the home region. Events add pressure on top of that normal traffic.

  1. 10:00 · Bad deploy

    A new version ships with a bug and some requests fail.

    Lasts until detected and repaired.

  2. 14:00 · Slow dependency

    An outside service slows down. Your servers fill with requests waiting on it.

    Added response time: ≈ 2,000 ms. Duration: ≈ 1,800 seconds.

  3. 19:00 · Viral spike

    A post blows up. Traffic climbs to several times normal within minutes.

    Traffic reaches 2 times the scheduled load; rises over 600 seconds, holds for 3,600 seconds and eases over 3,600 seconds.

What a strong design has to handle

  • The model and the feature store are on the path of every payment. What happens to your servers when one of them gets slow?
  • Make time-bounded decisions with versioned features and a safe model rollback.
  • A timeout must sit well inside the budget: with 150 ms for the whole check, hops that run side by side each give up within 50 to 100 ms. Some checks then fail fast when a helper is sick; the alternative is every server frozen waiting and the card network giving up first.
  • A circuit breaker in front of the decision store trades a short burst of fast failures for the whole fleet staying healthy, and the short timeouts do the same job for the model and the feature store. The business then decides what a failed check does: approve small amounts, decline large ones.

Make a prediction, run a full traffic day, then compare the result. The in-game editorial opens after a finished day and includes measured reference designs.

Concepts to practise

Related problems