Last night was rough for Inkwell Blogs. An app server died a few minutes before midnight, and for as long as it was down, a third of all requests failed. Then the database's primary went down, and nobody could publish anything until someone woke up and fixed it. The founders want Blogs to ride out both on its own.

What it does

• Read: someone opens a blog post.
• Write: a writer publishes or edits one.

Workload

A steady 1 500 rps, 90% reads / 10% writes, for 40 seconds, against 100 000 posts. This run replays last night:

• App replica 0 crashes at 5 s and is down for 10 s.
• The database's primary crashes at 20 s and is down for 15 s.

How to build it

The app tier is fixed at three medium replicas, and the database at the large tier with one read replica, copying each write to its replica about 20 ms after it says "saved". The crashes come with the components.

1. Build client, load balancer, app server and database, and run it. Look at the errors over time.
2. Make the load balancer notice a dead replica: health checks.
3. Make the database survive its primary: automatic failover promotes the read replica (it takes about 10 seconds).
4. Optional, for the hard tier: let the clients retry the requests that fail while things recover. Keep the backoff short enough that nobody is still waiting long after the database is back.