Inkwell Chat lets readers and writers message each other, one to one and in groups. It looks like "send a message", and it's hard for two reasons that pull in different directions: millions of people hold a connection open, and every message has to arrive once, in order, and survive a failure. The lesson follows the interview order, and the deep dives list the chapters each answer leans on.
Ask before you draw
| Question | Why it matters | Inkwell Chat |
|---|---|---|
| Who talks to whom? | One to one and groups are different loads | Direct messages and groups of up to a few hundred |
| How fast must it arrive? | Sets the delivery budget | Within a second for someone online |
| What if the recipient is offline? | Decides what must be stored | Kept, and delivered when they return |
| What order? | Order across a whole system is expensive | In order within one conversation, not across them |
| What can go wrong? | A late message is annoying, a lost one is a bug | Never lost, never shown twice |
Write the answers down. "In order within a conversation" is the sentence that decides how messages are stored and read.
Estimate the load
Work the chain yourself first, from the assumptions in the worksheet. The worked numbers open once you've done it.
The simulator runs a much smaller version with the same shape: 6 000 people connected and 300 messages a second.
API and data model
Two paths cover it. A person connects, with a WebSocket to a gateway, and holds the connection. A message is a call, POST /messages with the conversation, the sender and an idempotency key, so a retry doesn't send twice. Everything after the call is asynchronous.
- Messages: one row per message, keyed by conversation id and a sequence number, so a conversation reads back in order. This is the history.
- Inboxes: per person, what hasn't been delivered or read yet. It's what a returning phone catches up from.
- Routing: per person, which gateway holds their connection now. Written when they connect, read to deliver.
- Presence: per person, whether they're online. It expires on its own, and can be wrong for a few seconds.
A message is acknowledged once it's on the stream, not once it's delivered, so a slow recipient never slows the sender.
The path of a message
Held connections on one side, an ordered pipeline on the other.
The stream is the spine. Keying it by conversation puts every message of a conversation on one partition, which keeps them in order. Each service reads the stream on its own: delivery looks up the recipient's gateway and hands the message over, history writes it down. A service that falls behind doesn't slow the other.
The last hop, from the delivery worker to the gateway that holds the recipient, is a lookup in the routing table and a message to that gateway. The simulator stops at the inbox, so the claims below are about everything up to it.
One huge conversation
Keying by conversation has a cost: a partition is only as fast as one worker, so a conversation's deliveries can never be faster than that. Usually it doesn't matter. When one group chat is far busier than the rest, it does.
The slow part of a big group is delivery, not order: each message has to reach every member. Ordering is cheap. Each conversation has one owner that gives every message the next number, and people don't type fast enough for that to be the limit. So number the messages first, then spread the deliveries: one job per recipient, on a second stream keyed by recipient, which many workers read at once. Each client sorts by the number, so the order holds however the deliveries arrive.
When delivery goes down
The stream is what makes an outage a delay, not a loss. Workers pull messages from it, so while every delivery worker is down nothing is handed out: messages wait on the stream, the lag grows, and when the workers return they read on from where they stopped. The setting that decides whether that works is how long the stream retains messages.
One gateway or many
A crashed gateway instance drops every connection it holds, and its people all come back at once. How many instances share the 6 000 decides how many are dropped and how big the queue to reconnect is.
Deep dives
| They ask | What to say | Chapter |
|---|---|---|
| A group has 100 000 members | Fan out on read: store the message once, and let each member's client pull it | Fan-out |
| The recipient is offline | Keep it in their inbox, and send a push notification | Queues and workers |
| A message is delivered twice | An idempotency key from the sender, and a message id the client dedupes on | Idempotency |
| Ordering across devices | A counter per conversation, kept by the conversation's owner, numbers each message, and every device sorts by it. A stream offset won't do: one partition holds many conversations, so one conversation's offsets have gaps, and a device can't spot a missing message or ask for everything after message N. Offsets also change if the topic is re-partitioned | Streams |
| A reader has a phone and a laptop | Each device keeps a cursor: the last message number it has, per conversation. On reconnecting it asks for everything after its cursor, so each device catches up on its own. Read receipts are one more cursor per member, the highest number they've read, synced to their other devices | Streams |
| Who is online | A presence key with an expiry, refreshed by heartbeats, and allowed to be stale | Real-time connections |
| The gateway crashes | Spread people over many instances, and have clients back off with jitter | Real-time connections |
Pick two and say which you'd cover first and why.
Say the tradeoffs
| Choice | One side | The other side |
|---|---|---|
| Big group delivery | On the conversation's partition: simple, one worker's speed | Numbered there, then fanned out by recipient: parallel, a second hop |
| Retention and attempts | Short: cheap, loses messages in an outage | Long and patient: nothing lost, later and costlier |
| Gateway instances | Few and large: fewer to run, a big blast radius | Many and small: a small blast radius, more to run |
| Group delivery | On write: fast to read, a copy per member | On read: one copy, every reader pays |
Commit to a stream keyed by conversation, retention longer than the longest delivery outage plus the time to catch up (a minute or more in production; the practice's crash needs less), patient retries, and several gateway instances. Then say what you'd watch: delivery lag per service, messages dead-lettered, the connections held per instance, and the handshake wait after a crash.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Persistent connections to a gateway | WebSocket gateways at Slack and Discord, and AWS API Gateway's WebSocket APIs |
| An ordered log per conversation | Kafka partitions keyed by conversation, or Amazon Kinesis shards |
| An inbox for offline users | The unread queue that a phone syncs from when it reconnects |
| Presence with an expiry | A key with a TTL in Redis, refreshed by heartbeats |
Your practice
Inkwell Chat holds about 6 000 people and takes 300 messages a second, and a gateway instance crashes mid-run. Messages are arriving late and some are lost, and the crash leaves people waiting. Fix both, and then run it for as little as you can.