Sometimes there is simply more traffic than the servers can take, and no way to add more in time. You can't serve everyone. What you can choose is whether the requests you do take are served well.
More traffic than capacity
New servers take time: a budget to approve, machines to start, a database that can't grow any further. Until then, the traffic is what it is. Two app replicas that take 2 700 rps between them can't serve 3 600 rps, however they're configured.
| Requests per second | |
|---|---|
| Arriving | 3 600 |
| The servers can take | 2 700 |
| Can't be served, whatever you do | 900 |
Those 900 will fail one way or another. The question is what happens to the other 2 700.
Queues absorb bursts
Each app replica keeps a queue: requests that arrived while every thread was busy wait there for their turn. When traffic jumps for a moment, the queue holds the extra requests and the servers work through them once the burst has passed. Almost nobody gets an error; some people wait a little longer.
Queues under overload
When the overload doesn't stop, the queue never empties. It fills up, and from then on every request waits behind a full queue. Two replicas with 2 000 places each hold 4 000 requests, and at 2 700 rps that's a wait of about a second and a half.
Callers give up after a second. So the servers spend all their time on requests whose callers have already left, and almost nobody gets an answer in time. The servers are busy the whole time, but their goodput, the answers someone actually received, collapses.
Shedding load
Load shedding means turning the excess away on purpose, as early and as cheaply as possible. A short queue does it: once it's full, a new request is rejected in a millisecond instead of waiting to time out. The servers only work on requests they can finish in time.
| Queue per replica | Longest wait at 2 700 rps | Under overload |
|---|---|---|
| 2 000 | about 1.5 s | almost everything times out |
| 500 | about 0.4 s | full speed, but slow |
| 20 | about 60 ms | full speed, and fast |
Each wait counts the requests already running too: a replica's 50 threads take their turns on 4 cores, so 20 in the queue wait behind 70.
A fast "no" is useful to the caller: it can show a message, try a fallback, or retry after a backoff. A slow "no" only wastes everyone's time.
Sizing the queue
A queue should never hold more than the servers can work through before callers give up: throughput times timeout is the hard ceiling. That's Little's Law from Scaling Out again (requests waiting = rate × wait): servers that finish 2 700 rps, for callers who wait 1 s, can usefully hold about 2 700 waiting requests across all replicas. Below that, it's a trade. A longer queue rides out longer bursts; a shorter one keeps the wait short when traffic stays high.
Beyond the queue
A short queue sheds whoever arrives while it's full: the newest requests are turned away, whoever they are. Real systems usually shed more carefully:
| Technique | What it does |
|---|---|
| Shed early | Reject at the load balancer, before any work is done |
| Priorities | Turn away background jobs and prefetches before users |
| Rate limits | Cap each client, so one busy caller can't crowd out the rest |
| Backoff | Clients that are turned away come back later, not at once |
All of them follow the same idea: decide quickly who won't be served, and serve everyone else properly.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Short queues and fast rejections | connection and queue limits in Envoy and NGINX, HTTP 503 and 429 responses |
| Priorities | shedding background and prefetch traffic first, as Google and Netflix describe for their services |
Your practice
A morning TV show featured Inkwell's daily prompt. All day, about a third more people open the app than the two app servers can take, and the new servers won't arrive until tomorrow. The app gives up after a second, and almost every request times out in a long queue. Change how many requests each replica lets wait, so the ones that get in are fast.