Design Inkwell Pay, the ledger behind reader tips. A tip is a few cents moving from a reader to a writer, and the ledger can't get it wrong: a tip that's acknowledged must not be lost, a tip must not be recorded twice, and the service has to keep working when the database's primary dies.
Last month the primary crashed for ten seconds. Some readers saw an error, some tips that had been confirmed were gone afterwards, and a few readers were charged twice.
What it does
- Write: a reader sends a tip, which is recorded in the ledger.
- Read: a reader opens a receipt.
Workload
A steady 300 rps for 30 seconds, half tips and half receipts, over 10 000 ledger entries. This run replays last month:
- The database's primary crashes at 8 s and stays down for 10 s. A replica can be promoted in 8 seconds.
What's fixed
The app servers and the database tier are fixed. A save takes about 25 ms with replication, and the app gives up after 60 ms. About 2% of answers never reach the reader: their phone drops off the network after the tip was saved, and the app tries again.
How to build it
You choose the database's replicas, how it commits and replicates writes, and whether it fails over on its own. And you choose the tippers' retry policy: select the tippers client and edit its timeout, its retries and its idempotency key.
Steps
- Build the path: both clients, load balancer, app server, database. Run it as it is.
- Look at the errors, the tips lost after the crash, and the tips recorded twice.
- Fix each one. Then make the wait for a tip as short as you can without losing any.