A single key can draw more traffic than any one machine can serve. This chapter follows one viral post's like counter: what it does to a database, how a split counter takes the writes, and how to show a number that changes every few milliseconds.

Hot keys

Partitioning showed a hot key from the outside: one key's traffic stuck on one node, whatever the partition count. This chapter looks inside it. Every like on that post updates the same counter, so spreading traffic across machines doesn't help: that record is a hot key.

A relational database protects each row with a lock while it's being written, so two writes to the same row take turns. Each turn is short, about a millisecond from the change to the commit, so one row takes several hundred writes a second. The viral post sends 2 000. Past what the row takes, writes queue behind each other, each one holding an app server thread while it waits.

The database's CPU is nearly idle; the lock is the limit.

A bigger database doesn't help: more cores don't make a turn any shorter, and the row still takes its writes one at a time.

Splitting the counter

A key-value store has the same limit. A key lives on one partition, and its writes are applied one after another: DynamoDB, for example, takes about 1 000 writes a second on one item, however much capacity the table has. More partitions spread different keys, never one.

When one counter outgrows what one key takes, split it, the fix Partitioning sketched. Store the count as several sub-counters under different keys (likes:post:1, likes:post:2...), add each like to a random one, and add them up when reading.

The trade is on the read side: each count now costs ten reads. That's fine when counts are read through a cache or the edge, and expensive when they aren't. The same trick works in a relational database, with one row per sub-counter.

Reading a busy counter

The store keeps copies of every partition, and the consistency chapter's choice applies to every read: eventual from any copy, or strong from the latest committed value.

A counter liked every 2 ms is almost never current on a copy that's even slightly behind, so eventual reads of the viral post are nearly always out of date. Whether that matters depends on how far behind is too far.

Caching numbers that never stop changing

Purging a cached copy on every change works when changes are rare. On a counter liked every 2 ms, a purge lands before nearly any fill completes, so the edge never holds the count.

The edge refreshes the count every 20 ms instead of on every like.

A short TTL with no purge (the CDN's ttl_only setting, from The Edge) turns "always stale" into "at most 20 ms behind", and the store sees one read per edge every 20 ms instead of one per view. Every extra layer that keeps a copy, an app cache behind the edge for example, adds its own lag on top.

Hot keys in a feed

A feed has its own hot key: the writer with millions of followers. Copying each post into every follower's feed (fan-out on write) turns one post from a writer with 10 million followers into 10 million writes. Building the feed when it is opened (fan-out on read) makes a reader who follows thousands of writers pay on every open. Most large feeds mix the two: ordinary writers are pushed, and the few very popular writers' posts are merged in at read time, so a popular writer's posts become one hot key read by many, which a cache handles well. The Fan-out chapter, later in this track, puts numbers on both.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

Your practice

Social Media Post Likes: keep the like button standing through the viral post, keep counts trustworthy, and make them fast worldwide.