Design Inkwell Reactions: the like button and the count under every post. Most posts get a like now and then. A few go viral, and the like count of one post is then updated hundreds or thousands of times a second. Last time that happened, the likes started failing.
What it does
- Read: a reader opens a post and sees its like count.
- Write: a reader likes a post, which has to be recorded. A count may be a moment behind, but a like is never lost.
Workload
8 000 rps for 20 seconds, jumping to 40 000 rps for five of them, 50% counts / 50% likes, over 100 000 posts. Popularity is extremely skewed: a few posts, the first 100 keys, take most of the likes, and the hottest post alone takes about a third of them. Counts are read with the same skew.
What's fixed
The app servers are fixed: sixteen large replicas, so the servers aren't what runs out.
How to build it
The counts live in a key-value store, and a key-value store gives each partition a budget, so one key can only take so many writes, and so many reads, a second whatever the store's size.
A queue with workers can take writes off the store. Its Combine writes to one key for setting makes all the writes to one key within a window into a single job, so a thousand likes in a second become one update. A connection's Routing section can limit it to writes of keys below a number, and the hot posts are the first keys.
Steps
- Build the path: client, load balancer, app server, key-value store. Run it as it is, and look at the like errors in the burst.
- Send the hot posts' likes through a queue and workers, and everyone else's straight to the store. Select the connection from the app server to the queue: its Routing section can limit it to writes of keys below a number.
- Choose the window, and size the workers and the store for as little as you can.