retrystorm · Learn · Interview practice · Glossary · Components

Home › Learn › Reactions

Chapter 41 · Case studies · 28 min

Reactions

Designing Inkwell Reactions end to end: a count that may lag but not lose a like, the numbers, one hot key, combining writes, deep dives and tradeoffs.

Inkwell readers can now like posts all over Inkwell and see the count under each one. Most posts get a like now and then, but last week a post went viral, far bigger than the one that once broke Blogs' like button, and thousands of readers liked it every second. The likes started failing, and the product team wants the count to keep up the next time.

Inkwell Reactions is the like button and the count under every post. It looks like the simplest feature there is, and it contains one of the classic problems: a single counter that thousands of people update at once. This lesson follows the interview order, with the deep dives naming the chapters they lean on.

Ask before you draw

QuestionWhy it mattersInkwell Reactions
Must the count be current?A count that lags is far cheaperA moment behind is fine
What can't go wrong?Decides what to protectA like lost, or counted twice
What does a reader see?A count on every post in a feed, and whether they liked itBoth
How uneven is it?One post can be a thousand times busier than mostA few go viral
Can a user take it back?Unlike must be as safe as likeYes

The second row is the sentence to write down: a count may lag, but a like may not be lost. It's what allows the cheap design later.

Estimate the load

Work the chain yourself first, from the assumptions in the worksheet. The worked numbers open once you've done it.

API and data model

PUT /posts/{id}/like records a like, and DELETE removes it. Both are safe to repeat. GET for a feed returns the counts together with whether the reader has liked each post.

  • Likes: one row per (post, user), with a unique key on that pair. It's the record, it's what makes a second like from the same user do nothing, and it answers "did I like this?".
  • Counts: one number per post, kept apart from the likes. It's a copy, it can lag a moment behind, and it can be rebuilt by counting the likes.

Keeping the count separate is the first big decision. Counting the likes of a post each time it's shown would be far too slow, and the count doesn't need to be current to the second.

When one post goes viral

A post's counter lives on one partition, and that partition has a write budget.

The simplest design writes every like straight to the counts store. It works until a post goes viral.

Combine the writes

The answer is to write fewer times, not to write faster. A like only changes the count by one, so a thousand likes in a second can be applied as one update of a thousand. A plain queue won't do it: SQS, RabbitMQ and Kafka hand each message on as it was sent. Something has to add them up. A stream processor such as Flink or Kafka Streams, or a buffer inside the app, sums the increments for each post over a window and writes once. The simulator models this as a setting on its queue, Combine writes: all the writes to one key that arrive within a window become a single job.

Hot posts are combined, everyone else is written straight through.

Only the hot posts need it, so the app sends writes for them to the queue and everything else direct. In the simulation the hot posts have the lowest numbers, so the line is a post number. Real systems find hot posts by counting, for example the posts that got more than a threshold of likes in the last few seconds.

The like rows of a viral post are hot too: 10 000 new rows a second for one post. Partition the likes table by user, not by post, or write the rows in batches, so they spread out. Only the count is one number, so only the count has to be combined.

A combined update is not safe to repeat. If a worker crashes after adding 1 000 and before the queue hears back, the job is offered again and the 1 000 is added twice. So the worker records the id of each batch it applied, next to the count, and skips a batch it has already applied.

Deep dives

They askWhat to sayChapter
A user double-taps the likeThe unique key on (post, user) makes the second do nothingIdempotency
A user unlikesDelete the row, and subtract from the count, combined the same wayIdempotency
The count drifts from the likesA background job recounts, and corrects the stored numberConsistency
Reading counts for a whole feedCache them with a short TTL, and read them in one batchCaching
Spread one counter over several keysSub-counters: write to a random one, add them up on readHot keys
A trending listLikes also go to a stream, and a windowed aggregation keeps the top postsStreams
Bots liking in bulkA per-user rate limit in frontRate limiting

Take two of these and justify the order.

Say the tradeoffs

ChoiceOne sideThe other side
Count updatesWith each like: always current, fails on a hot postIn batches: a moment behind, rebuildable from the likes
Hot postsDirect writes: simple, fail when a post goes viralCombined per window: absorbs it, counts lag
WindowShort: fresher counts, more writesLong: fewer writes, staler counts
Hot counterSub-counters: more writes spread out, a read costs moreCombining: one write, one read, a delay

Commit to a likes table with a unique key, a count in a key-value store that lags by about a second, and the hot posts' writes combined for a second. Then say what you'd watch: failed likes, the age of the oldest combined job, and how far the counts drift from a recount.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

IdeaWhere you'll meet it
A counter with atomic incrementsRedis INCR, DynamoDB atomic counters
A hot key throttled on one partitionDynamoDB's per-partition limits and hot-key throttling
Aggregating writes before storing themWindowed aggregations in Kafka Streams and Flink
Finding the hot keys cheaplyA count-min sketch, which counts approximately in small memory

Your practice

A post went viral and the likes started failing. Keep the likes safe and the counts close behind, and then make it cost as little as you can.

Key takeaways

  • A count may lag, but a like may not be lost.
  • The averages are easy, and one viral post on one key is the problem.
  • A key lives on one partition and takes only so many writes a second, whatever the store's tier or partition count.
  • Combine the hot posts' likes into one update per window: fewer writes, slightly staler counts.
  • Commit to a likes table, a count that lags, and the hot posts combined, and watch for failures and drift.

Read first

  • Streams
  • Rate Limiting
  • Data Modeling
  • The Design Interview

Practise it

  • Reactions: Design Inkwell Reactions: likes and counts that survive a post going viral.

Terms

  • Write coalescing: Combining the writes to one key that arrive within a short window into a single write, so a hot key takes far fewer updates at the cost of a short delay.
AI system design mock interview · About · Contact · Privacy · Terms of use