A load balancer does more than spread requests around. How it picks a replica decides whether a few slow requests stay a small problem or slow everyone down.
What the load balancer decides
For every request, the load balancer picks one replica. It has to decide fast, usually without knowing how long the request will take. The four common strategies use different information:
| Strategy | Picks | Knows about |
|---|---|---|
| Round robin | the next replica in turn | nothing |
| Random | any replica | nothing |
| Least connections | the replica with the fewest requests in flight | how busy each replica is right now |
| Consistent hash | the replica that owns the request's key | which key the request is for |
Same replicas, same traffic. Only the choice changes.
When every request costs the same
If every request takes about as long as the others, round robin is hard to beat. Each replica gets exactly its share, in a steady rhythm, and none of them builds a queue while another sits idle.
Random sounds just as fair, and on average it is. But "on average" hides streaks: by chance, one replica sometimes gets three requests in a row while another gets none. Those short pile-ups nudge the median up a little and push the tail up a lot.
When some requests are slow
Real requests are rarely equal. Opening a note takes a few milliseconds; exporting a whole notebook to PDF can take a hundred times longer. Round robin doesn't know that. It keeps handing requests to a replica whose cores are all busy with exports, and everything queued behind them waits.
Least connections looks at how many requests each replica is working on and picks the least busy one. Slow requests still take long, but they stop dragging fast requests down with them.
Consistent hashing and hot keys
Consistent hashing sends every request for the same key to the same replica. That's useful when replicas keep something per key, like a local cache or other per-key state: the next request finds it already there. It also means adding a replica moves only a small share of keys.
The catch is skew. If one key is far more popular than the rest, its replica gets far more than its share, and no other replica can help.
Balancing doesn't add capacity
A smarter algorithm only moves work around. If the replicas together can't keep up, every strategy fails the same way: queues fill everywhere, and requests are turned away whichever replica they were sent to.
Pick the algorithm for how requests differ from each other; add replicas for how many there are.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Round robin and least connections | NGINX (round robin by default, least_conn), HAProxy (leastconn), AWS ALB (least outstanding requests) |
| Consistent hashing | Envoy's ring hash and Maglev balancers, memcached clients (ketama) |
Your practice
Inkwell now lets writers export notebooks to PDF, and a few of those exports take far longer than anything else. The servers aren't full, yet the slowest requests keep getting slower. With the same four replicas, get 99% of requests under 125 ms.