Design Inkwell Chat, direct and group messages for readers and writers. Two things have to work at once:

• Staying connected: about 6 000 people keep a connection open so messages reach them the moment they arrive. When a gateway instance crashes, its people reconnect together.
• Delivering messages: every message is acknowledged once it is safely on the stream, stored for history, and delivered to its recipients, in order within a conversation, and never lost.

What it does

• Connect: a person opens a connection and holds it.
• Write: a message is sent, appended to the stream, stored, and delivered.

Workload

6 000 people connect and stay for about 30 seconds each, and 300 messages a second are sent for 40 seconds across 1 000 conversations. A gateway instance crashes at 20 seconds for 4 seconds. A delivery or a store takes a worker about 20 ms, and about 2% fail and are retried.

What's fixed

The app servers are three medium instances, and each key-value store is medium with four partitions. A handshake takes about 10 ms of one of a gateway instance's cores.

How to build it

As shipped, the stream has four partitions and keeps messages for three seconds, one gateway instance holds every connection, and a failed delivery is retried after a second. Messages arrive late, some are lost, and a crash leaves people waiting for a connection.

The stream, its two services deliver and store with their workers, the gateway gw, and both clients are already on the canvas: people hold connections, and senders send messages. You choose the stream's partitions, retention and retry settings, the gateway's tier and instances, and each pool's size.

A stream gives a group its messages in order within a partition, so a conversation's key keeps its messages in order, a partition is only as fast as one worker, and workers beyond the partition count have nothing to do. Messages are kept for the retention window, not until they are read. A gateway holds a fixed number of connections per instance, and a crash drops every connection on the instance.

Steps

1. Build the paths: people to edge to the gateway, and senders to lb to an app server to the stream. Connect the stream to both services, and the services to a key-value store each. Run it as it is.
2. Look at the connections held, the handshake wait, each service's lag and the messages lost.
3. Fix each cause, then give each part only what it needs.