Inkwell's live comments keep a connection open from every reader's browser, so new comments appear the moment they're written. Right now there are about 6 000 readers connected at any moment, through one gateway. Last week the gateway crashed for a few seconds, every reader was dropped at once, and they all tried to come back together. The gateway spent the next quarter of a minute setting up connections, and some readers waited far longer than that.
What it does
- Connect: a reader opens the page and connects. They stay connected for about 30 seconds, then leave, wait about 3 seconds, and connect again.
Workload
6 000 readers for 60 seconds. A gateway instance crashes at 20 s and is down for 4 seconds, taking its connections with it.
How to build it
The session length, the handshake time and the crash are fixed. A handshake takes about 10 ms of one of an instance's cores, so an instance can only accept so many new connections a second. You choose the gateway's tier and how many instances it has, how often a connected reader sends a heartbeat, the load balancer, and the readers' retry policy: select the readers client and edit its retries and backoff.
- Build the path: readers, load balancer, gateway. Run it as it is.
- Look at how long a connect waits, and how many readers are connected at the end.
- Make it recover quickly, for as little as you can.