Every weekday at 8:00 Inkwell sends a writing prompt, and a few seconds later a wave of users opens the app at once. For three seconds there's more traffic than the two app servers can take, and they turn the extra requests away.
The app gives up on a request after 300 ms and retries right away, twice. The rush still ends with an error for about one request in eight.
What it does
- Read: a user opens today's prompt or a note.
- Write: a user saves a note.
Workload
1 400 rps for most of the 15 seconds, jumping to 4 000 rps for three seconds, 90% reads / 10% writes, against 10 000 notes.
How to build it
The servers are fixed: two app replicas behind a load balancer, and an xlarge database. What you can change is the app's retry policy: select the client and edit its timeout and retries.
- Build the path: client, load balancer, app server, database. Run it as it is.
- Select the client. How many tries does it make, and how long does it wait between them?
- Change the policy until the rush passes without errors, and nobody waits much longer than the rush itself.