Inkwell Blogs paged the on-call engineer: about one request in four fails during the evening peak. Dashboards show the error rate, and nothing else makes it obvious why. Looking closer, the errors don't all say the same thing.
What it does
- Read: a reader opens a post, with its related posts. The related-posts query is slow, about 120 ms, mostly waiting on disk.
- Write: an author edits a post.
Workload
A steady 2 000 rps, 95% reads / 5% writes, for 40 seconds, over 100 000 posts. This run replays the evening:
- App replica 1 crashes at 5 s and is down for 10 s.
How to build it
The app servers are medium replicas behind a load balancer, and the evening ran on four. You choose how many replicas, the load balancer's health checks, and the database's tier and read replicas. Each database node accepts a limited number of connections. Reads go to the read replicas only, so it's the replicas' connections that the reads share:
| Tier | Connections per node | Price per node |
|---|---|---|
| Small | 100 | $0.12/h |
| Medium | 200 | $0.30/h |
| Large | 400 | $0.70/h |
| XLarge | 800 | $1.50/h |
- Build the path: client, load balancer, app server, database. Run it as it is.
- Open the results and look at the failures by reason, and at which component refuses requests. Different components refuse requests for different reasons.
- Fix each cause, for as little as you can: a crash shouldn't be noticed by readers.