When a writer publishes a post, several things should happen: followers hear about it, search learns of it, analytics counts it. A job queue hands each job to one worker, so it can't do that. This chapter is about sending one event to several services, and what each of them needs to keep up.
One event, several subscribers
Publish/subscribe (pub/sub) sends each published message to every subscriber. Each subscriber gets its own copy and works through its copies at its own pace. The publisher doesn't do the work and doesn't wait for it: it publishes once and the message is on its way.
A topic is the component that does this. It stores each message once, acknowledges the publisher, and puts a copy on every subscription's own backlog. A subscription is one subscriber's connection to the topic: its worker pool, its backlog and its own retry settings. Amazon SNS feeding an SQS queue for each subscriber works the same way.
One post, three copies, three backlogs.
Adding a subscriber means connecting another worker pool to the topic. Nothing about the other subscribers changes, and the publisher doesn't need to know.
A slow subscriber only hurts itself
Because each subscription has its own backlog, a subscriber that can't keep up builds a backlog of its own. Everyone else carries on.
How far behind a subscriber is, is called its consumer lag: the number of messages waiting, or the age of the oldest one. It's the number to watch and alert on for each subscriber, since the publisher's own latency won't show it.
Size each subscriber for its own deadline
Subscribers don't all need to be fast. Followers want a notification within a second, search can take a few seconds, and analytics can be a minute behind. A subscriber needs enough workers to keep up with the rate of messages: at 90 posts a second, 100 ms of work each is 9 cores busy, so three medium instances (four cores each) with headroom. Beyond keeping up, more workers only buy a shorter wait for each message. A subscriber's time to finish counts from the moment a message is published to the moment its worker is done, so the wait in the backlog is part of it.
A subscriber that can't catch up has another choice: a message TTL. A copy that has waited longer than the TTL is dropped instead of delivered late. That suits data that's worthless when stale, like a live count, and it loses those messages for that subscriber alone: the others still get theirs, and the publisher isn't told.
Topic or log
A plain topic deletes a subscriber's copy once that subscriber has handled it. (Some, like Google Cloud Pub/Sub, can be set to keep handled messages for a while and replay them.) A log keeps every message, in order, for days, and each consumer keeps a position: the next message it will read.
The next chapter, Streams, covers logs in full. Here three services read the same 30 messages:
Keeping history gives you two things a deleted-on-delivery topic can't: a new service can be filled with everything that happened before it existed, and a consumer that had a bug can go back and read again. The price is storage and operating the log: its partitions, its retention and its consumers' lag, which the Streams chapter covers. Kafka and Kinesis are logs; SNS with SQS is a topic with a queue for each subscriber.
Each subscriber gets at least one copy
Delivery to each subscriber is at least once, so a subscriber can see the same message twice, after a crash or a slow acknowledgement. Each subscriber has to cope with that on its own, with an idempotency key or an operation that's safe to repeat, as the idempotency chapter described. Each one also needs its own dead-letter queue for messages it can't process: after a few attempts the poison message is set aside for someone to look at, instead of being retried forever, and only that subscriber's copy is. The other subscribers handled theirs long ago.
Order is weaker still: there's no global order across subscribers, and by default a topic gives no order within one subscriber either, unless you turn ordering on (Google Cloud Pub/Sub's ordering keys, SNS FIFO topics). A log keeps order within a partition. If a subscriber needs "created" before "deleted", say so in the design.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| A topic fanning out to queues | Amazon SNS to SQS, Google Cloud Pub/Sub subscriptions |
| A log with consumer positions | Kafka consumer groups, Amazon Kinesis |
| Consumer lag as a health signal | Kafka consumer lag, the SQS age of oldest message metric |
| Dead-letter queues per subscriber | SQS redrive policy, Pub/Sub dead-letter topics |
Your practice
When a writer publishes, followers must be notified, the post indexed and the count updated. Each service has its own deadline. Size each one for its own, and pay for no more.