After the 8:00 rush Inkwell's servers don't recover. Their queues hold 2 000 requests each, the app gives up on a request after 300 ms and retries, and every retry lands behind requests whose callers have already left. For the rest of the morning most requests fail.
You can't shorten the queues or add servers. What you can do is make the app stop calling when the calls are clearly failing, and start again when they work.
What it does
- Read: a user opens today's prompt or a note.
- Write: a user saves a note.
Workload
1 400 rps for most of the 15 seconds, jumping to 4 000 rps for three seconds, 90% reads / 10% writes, against 10 000 notes.
How to build it
The servers are fixed: two app replicas with deep queues behind a load balancer, and an xlarge database. Build the path, then add a circuit breaker between the client and the load balancer.
- Build the path: client, load balancer, app server, database. Run it as it is.
- Add a circuit breaker in front of the load balancer and run it again. It is open for a while and refuses requests at once, so some fail.
- Select the breaker. How many failures does it wait for, and how long does it stay open? Tune it until fewer requests fail and the slowest ones stay quick.