Every response has a size, and every server can only send so many bits a second. Small responses never notice; large ones can fill a server's network long before its CPU is busy.
Responses have a size
A note is a couple of kilobytes. A picture is hundreds of kilobytes, a photo a few megabytes, an exported book much more. The server has to send every byte of every response, so a server sending pictures does far more work on the wire than one sending notes, for the same number of requests.
| Response | Size |
|---|---|
| A note | 2 KB |
| An attachment or profile picture | 300 KB |
| A photo | 3 MB |
| A notebook exported to PDF | 20 MB |
Bandwidth
A server's network card sends a fixed number of bits a second: its bandwidth. The time to send a response is its size over the bandwidth, and the most responses a server can send each second is the bandwidth over the size.
| Response size | Time to send at 2.5 Gbps | Most per second |
|---|---|---|
| 2 KB | 0.006 ms | about 150 000 |
| 300 KB | 1 ms | about 1 000 |
| 3 MB | 10 ms | about 100 |
Count bits, not bytes: a byte is 8 bits, so 1 000 responses of 300 KB a second is 2.4 Gbps. Cards are rated in bits, sizes in bytes, and mixing them up is an easy factor of eight.
A full card
When responses need more bandwidth than the card has, they queue to be sent, just as requests queue for a busy CPU. The card is full, waits grow, and requests start failing, while the CPU sits almost idle.
A CPU graph won't show this. Watch the network as a limit of its own, like CPU and threads.
Big responses hold back small ones
Every response a server sends shares its card. Responses go out as packets, and packets from many responses interleave, so a 2 KB note doesn't wait for whole files to finish: each response in flight gets an equal share of the card. While the card has room, that costs the note almost nothing. At 90% busy, every response takes about ten times as long as on an idle card, and ten times a 2 KB send is still well under a millisecond. Once the attachments ask for more than the card can send, though, they arrive faster than they finish. The responses in flight pile up, each one's share keeps shrinking, and the note, sharing with all of them, slows down and times out with them. Once large files share a server with small requests, the small requests are only as fast as the large ones let them be.
More bandwidth
Bandwidth comes with the machine. A bigger tier has a faster card (1, 2.5, 5 and 10 Gbps from small to xlarge), and every replica brings its own.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Network per machine | the "network performance" column of cloud instance types, such as "up to 10 Gigabit" |
| Paying for bytes | cloud egress pricing, charged per GB sent out |
Your practice
Writers can now attach files to their notes, about 300 KB each. Since attachments launched, even plain notes have slowed down, and the database's CPU is barely busy. The app tier is fixed. Choose the database's tier and read replicas so it has the bandwidth to send everything, for as little as you can.