Everything readers do on Inkwell, a post, a like, a follow, is an event, and the events are appended to a stream. Three services read it:
- Indexer: keeps search fresh, and needs each event within about a second.
- Analytics: counts events, and can be a few seconds behind.
- Recommender: a new service that starts reading ten seconds in, from the beginning of everything the stream still holds, and has to catch up.
One writer is very noisy: a viral account posts and reacts all day. As shipped, the stream has four partitions, keeps events for three seconds, and a failed event is retried after a second. Search is slow, events vanish before analytics reads them, and the recommender never catches up.
What it does
- Write: an event is appended to the stream, and each service reads it.
Workload
A steady 400 events a second for 20 seconds: 392 a second from readers, spread evenly over 10 000 keys, and 8 a second from viral, always the same key. Each event takes a service's worker about 20 ms and a write to the key-value store. About 2% of event handlings fail and are retried.
How to build it
The stream and a worker pool for each service are already on the canvas: events, with indexer, analytics and recommender. The recommender's late start is fixed. You choose the stream's partitions, retention and retry settings, and each pool's size.
A stream gives a group its events in order within a partition, one at a time, so a partition is only as fast as one worker, a failed event holds back everything behind it in its partition, and workers beyond the partition count have nothing to do. Events are kept for the retention window, not until they are read.
- Build the path: both clients, load balancer, app server, stream. Connect the stream to the three workers, and the workers to a key-value store. Run it as it is.
- Look at each service's lag, and at the events lost.
- Fix each cause, then give each service only what its own deadline needs.