Inkwell readers can now tip the writers they like. A tip is saved by the database, which usually takes about 100 ms, and about one save in seven takes longer than 300 ms. The app gives up after 300 ms and tries again, up to three tries in all. And about 2% of answers never make it back: the reader's phone drops off the network after the tip was saved, so the app sees a failure and tries again. Support is getting emails: some readers were charged twice for one tip.
What it does
- Read: a reader opens their tip receipts.
- Write: a reader sends a tip, which is saved once and charged once.
Workload
A steady 300 rps for 2 minutes, 70% reads / 30% tips, against 10 000 receipts.
How to build it
The servers are fixed: three app replicas behind a load balancer, and an xlarge database. What you can change is the app's retry policy: select the client and edit its timeout, its retries and its idempotency key.
- Build the path: client, load balancer, app server, database. Run it as it is, and look at how many tips were applied twice.
- Find the setting that makes a retry safe.
- Then keep failures and waiting low: a tip shouldn't fail just because the database was slow once.