Inkwell Blogs now fails over on its own: when the primary dies, a read replica takes over within seconds. But after last week's failover, a few dozen writers found that posts they had saved, and been told were saved, were gone. They were the last few milliseconds of writes, still on their way to the replica when the primary died.
What it does
- Write: a writer saves a post. Once Inkwell says "saved", it must never be lost.
- Read: someone opens a post.
Workload
A steady 2 000 rps, 50% reads / 50% writes, for 30 seconds, against 10 000 posts. The primary crashes at 10 s and stays down for 18 s; automatic failover promotes a replica after about 10 s.
How to build it
The app tier is fixed, and the database is large, with two read replicas and automatic failover. You choose how the database makes a write safe before it answers:
- Commit mode: whether the write must be on the primary's disk (
sync) or only in its memory (async) before "saved". - Replication: whether the write must also have reached a replica (
sync) or not (async) before "saved".
- Build client, load balancer, app server and database, and run it. Look at Lost writes.
- Find the settings where no saved post is lost, and see what they cost in write latency.