Everything so far has been a request and a response: the client asks, the server answers, the connection is done. Live comments, chat and notifications need the server to speak first. That means connections that stay open, and a different set of limits: how many can be held, how fast new ones can be set up, and what happens when thousands of them come back at once.
Pushing to a client
A server can't open a connection to a browser behind a firewall, so the client always connects first, and the question is how to keep that connection useful.
| Method | How it works | What an idle user costs |
|---|---|---|
| Polling | The client asks "anything new?" every few seconds | A request every few seconds, nearly all of them empty |
| Long polling | The client asks, and the server holds the request until there's news | An open request, which on a thread-per-request server holds a thread |
| Server-sent events | One long HTTP response the server keeps writing to | An open connection, one way only |
| WebSocket | One connection that stays open, both ways | An open connection |
One connection per reader, held by a gateway built for it.
Polling is simple and wasteful: ten thousand readers polling every two seconds is 5 000 requests a second, almost all of them answering "nothing". The other three keep a connection open and only spend work when something happens.
A connection is a slot, not a thread
An open request on a thread-per-request app server, like the ones in this course, holds a thread for as long as it waits, and an app server has a few dozen to a couple of hundred of them. Event-loop servers (Node.js, Netty) hold far more, since a waiting request is only a little memory. A connection gateway is built on that idea: it holds each idle connection in a slot, a small piece of memory with no thread of its own, so one instance holds tens of thousands.
Everyone reconnects at once
Setting up a connection costs work, the handshake: the transport and the login check, a few milliseconds on one of the instance's cores. An instance can only do so many a second. With a 10 ms handshake on four cores that's about 400 new connections a second, and 2 000 readers connecting together wait up to five seconds for the last one.
That's tolerable at startup, and it's what happens after a crash. A gateway restarts or fails, every connection it held drops together, and the clients all retry.
The cost of staying alive
A connection that's silent could be healthy or dead: a phone that went into a tunnel doesn't announce it. So the client and server exchange a small heartbeat every so often, and a connection that misses a few is closed. Each heartbeat is a little work on the gateway's cores, for every connected client.
More than one gateway
One instance isn't enough, and a crash makes it a single point of failure, so there are several. A load balancer spreads new connections, and least connections suits it better than round robin, since sessions last very different times. Watch it after a restart, though: the instance that just came back holds the fewest, so every reconnect goes to it at once, and its cores become the bottleneck. Balancers ramp a returning instance up slowly for this reason. Rolling out a new version drops its connections, so deploy one instance at a time and drain it first: tell its clients to reconnect, spread over a minute.
Spreading connections creates a new problem. To send a message to a user, something must know which gateway holds that user's connection. That's a routing table, often in a key-value store: when a user connects, their gateway writes itself there, and when a message arrives, the sender looks the gateway up and passes the message on. It's the same store that answers "is this user online?", and it is written to on every connect, which is another reason a mass reconnect is expensive.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Persistent, two-way connections | WebSockets, defined in RFC 6455, and libraries like Socket.IO |
| One-way streaming over HTTP | Server-sent events, used to stream live updates and generated text |
| A gateway holding connections | The gateways of Slack and Discord, and AWS API Gateway's WebSocket APIs |
| Heartbeats and idle timeouts | WebSocket ping and pong frames, and load balancers' idle timeouts |
Your practice
Inkwell's live comments hold about 6 000 readers connected. When a gateway instance crashed, everyone came back at once and the queue for a connection ran to many seconds. Make it recover quickly, and then make it cost as little as you can.