A queue forgets a job once it's done, and a topic forgets a copy once a subscriber has handled it. A stream remembers. It's a log of messages, in order, that stays where it is while any number of services read it, each at its own position. That one difference brings ordering, replay and a new way to fall behind. This chapter is about the stream, and what its partitions, its order and its retention mean for the services reading it.

A log, not a queue

A stream is an append-only log split into partitions. A message is appended to one partition, chosen by its key, and stays there until the retention window passes, whether or not anyone has read it. A consumer group is a service reading the stream, and it has a position in each partition: the next message it will read. Two groups never disturb each other.

Each group reads every message at its own pace, and the stream keeps them.

Partitions set the parallelism

A group reads one partition's messages strictly in order, one at a time. That's what makes the order mean something, and it makes a partition exactly as fast as one worker: if handling a message takes 20 ms, a partition delivers 50 a second, however many workers the group has. The trip to the stream barely counts: a worker fetches a batch of messages at a time, up to hundreds, and works through them before asking again. The parallelism a group can use is the number of partitions. Kafka 4's share groups trade the order away to let several workers take one partition's messages, the way a job queue does.

Here is the same idea with a few messages you can watch. Forty events arrive over ten ticks, and each consumer handles one a tick:

Order, and the hot key

A message's key decides its partition, so all the messages for one key land in one partition and arrive in the order they were sent. That's the guarantee: order for a key, and no order between keys. It holds while the partition count stays put: the partition is a hash of the key modulo the count, so adding partitions moves most keys, and for a while a key's new messages sit in a different partition from its old ones. Pick the count up front, with room to grow.

The cost is the hot key from Partitioning and Hot Keys again, with order now at stake: a key can't be split. A viral post, a busy user or one noisy customer sends all its messages to one partition, which is as fast as one worker, however many partitions there are.

Positions, crashes and re-delivery

A group remembers its place by committing an offset, which says "everything before this one is done". The committed offset is what survives a crash. When a consumer dies, the group rebalances: the partitions are handed out again (with the eager assignors, long Kafka's default, every consumer gives up all its partitions and the whole group stands still while that happens; the cooperative sticky assignor moves only the partitions that change owner), and the dead consumer's partitions resume from their last committed offset, not from wherever it had got to. Consumers still alive repeat nothing if they commit their place as they hand a partition over, which the app does in a rebalance listener; it isn't automatic.

That makes the moment of the commit a choice:

Committing less often costs less, and repeats more after a crash. Most systems commit after the work and make the work safe to repeat, as the idempotency chapter described. That is how you get effectively once: repeats happen but change nothing, either because the work is idempotent or because the offset is committed in the same transaction as the output (Kafka transactions do this when the output is written back to Kafka).

A failed message holds the line

Because a partition goes in order, a message that fails stops the partition, and a consumer handles its messages one at a time, so it stops everything that consumer reads. The group retries it, and nothing behind it moves until it succeeds. If it runs out of attempts it's set aside (dead-lettered) and the line moves on.

Retention, and falling too far behind

The stream keeps messages for the retention window, not until they're read. A group that stays behind longer than the window finds the messages it hasn't read are gone, and they're lost to it.

Replay, and a new reader

Because the stream keeps its messages, a new group can start at the beginning of what's kept, and read everything that came before it existed. It's how a new service is filled with history, and how a service with a bug reads again after a fix.

Counting in windows

Streams also feed live counts: likes per post in the last five minutes, the most-read pages this hour. A consumer that counts keeps a total per key for each window of time. A tumbling window is a fixed slice (every five minutes, one after the other); a sliding window moves with the clock. Events arrive late, since a phone that was offline sends its likes a minute after the fact, so each event carries its event time (when it happened), apart from the processing time (when the consumer sees it). A watermark is the consumer's running estimate of how far event time has got: "nothing older than 12:04 is still on its way". A window closes when the watermark passes its end, plus any allowed lateness you choose, and events later still are counted into a correction or dropped.

The top K of a window, say the ten most-liked posts, is simple when the counts fit in memory: a count per key and a heap of the largest K. Across partitions, key the events by post, so each post's whole count lives on one partition; then each partition sends its own top K and one place merges them. Keyed any other way, a post's count is split over partitions, and a merge of partial top-K lists can miss it. When even the counts don't fit, a count-min sketch keeps approximate counts in a small fixed table: it can count a key a little high but never low, which is fine for a trending list. It can only answer "how many for this key?", not list the keys, so pair it with a small heap of the current top K, checking each event's key against the sketch as it arrives.

Queue, topic or stream

Use a job queue when work needs doing once, in any order, and nobody needs the message afterwards. Use a topic when several services each need their own copy of an event and nothing else. Use a stream when the order of events for a key matters, when services should be able to read at their own pace, or when a new service has to see history. The price is partitions to size, retention to set, a hot key that can stall a partition, and a failed message that holds a line.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

In the simulator, one partition takes as many writes as you send it. A Kinesis shard takes 1 000 records or 1 MB a second, and a Kafka partition tens of megabytes a second, so on Kafka it's usually the consumers, not the writes, that decide how many partitions you need.

Your practice

Inkwell's events feed search, analytics and a new recommender. Search is slow, events vanish before analytics reads them, and the recommender never catches up. Fix all three, and then give each service only what its own deadline needs.