The chapter on hot keys sketched two ways to build a feed from the posts of the writers a reader follows. This chapter puts numbers on them, and finds out what the push way costs when a writer has a lot of followers.
Build the feed when it's read
Somebody has to put each post in front of all of its readers, and the only choice is when to do the work. The simplest feed is built when a reader opens it: look up everyone they follow, fetch each writer's latest posts, merge them by time and show the top of the list. A post is written once, and every reader pays for it when they look. This is fan-out on read, or the pull model.
Pull: one write per post, but a feed read touches every writer the reader follows.
It's cheap for writers and slow for readers, and readers vastly outnumber writers: 600 feed opens a second, each reader following 200 writers, is 120 000 lookups a second.
The opposite is to do the work ahead of time. When a writer posts, copy the post into a ready-made feed for each of their followers, so opening a feed is a single lookup. This is fan-out on write, or the push model.
Push: a read is one lookup, but every post becomes a write per follower.
Push moves the cost from the reader to the writer. That's a good trade when reads are much more common than posts, which is the usual case for a feed.
Pushing costs a write per follower
Fan-out on write multiplies writes. The amount of feed writing the system must absorb is posts per second times followers per post: 40 posts a second to 500 followers each is 20 000 feed writes a second, from a service that only sees 40 requests a second of writing.
You wouldn't do all that inside the request: a writer shouldn't wait for 500 copies. The post goes to a queue (the one from the queues and workers chapter) as one job, and in the background a worker pages through the writer's followers, writing the post's id into their feeds a batch at a time. The writer is told "posted" after one fast append. The experiments here count each feed write as its own job; the work is the same, a write per follower. What changes for the reader is how fresh the feed is: how long after a post it shows up in followers' feeds.
The star problem
An average hides the tail. If most writers have 30 followers, 30 feed writes a post is easy. Then one writer with 20 000 followers posts, and a single click makes 20 000 feed writes, queued ahead of the posts of everyone who published a second later. The feeds of ordinary readers go stale because of one account.
Push for most, pull for stars
Real systems combine the two. Ordinary writers are pushed into their followers' feeds. Writers past a follower threshold, say a few thousand, aren't pushed at all. When a reader opens their feed, it's read from the precomputed list, and the latest posts of the few stars they follow are fetched and merged in on the spot.
| Push (fan-out on write) | Pull (fan-out on read) | |
|---|---|---|
| Work happens | When a post is published | When a feed is opened |
| Cost grows with | Followers per post | Writers followed per reader |
| Reading a feed | One lookup, fast | Many lookups, slower |
| Feed freshness | Delayed by the queue backlog | Always current |
| Best for | Many readers, ordinary writers | Writers with huge audiences |
Fetching the star's posts at read time costs every reader a little more, a merge the experiment puts at about 4 ms (24 ms instead of 20). In exchange it removes the 40 000 feed writes a second that the star's two posts a second caused, which otherwise slowed every reader to about 40 ms and left the feeds about 16 seconds behind. The threshold is a tuning knob. Lower it and more writers count as stars: fewer copies are written, and more feeds gather posts when they're opened, so reads get slower. Raise it and the reverse.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Fan-out on write, per-user feed lists | Twitter's home timeline, kept in Redis lists, described in its engineering talks |
| Hybrid push and pull | Twitter handling accounts with very many followers separately; a standard answer to "design a news feed" |
| One event, many consumers | Amazon SNS to SQS fan-out, several Kafka consumer groups reading one topic (within a group, consumers share the work) |
| Backlog age as a health signal | SQS "age of oldest message" metric, Kafka consumer lag |
Your practice
Inkwell readers follow writers now, and each post is copied into the feed of its 30 or so followers. Size the workers and the database so a post reaches the feeds within a few seconds without paying for workers nobody needs.