Inkwell's backend works for a few hundred beta testers. Tomorrow the press arrives with thousands more. This chapter is about why designs stop working as traffic grows, and what to do about it.

Every server has a limit

An app server has a few CPU cores and a limited number of threads, slots for requests in progress. Each request needs some CPU time, and it holds a thread until it is completely done, including the time spent waiting for the database.

That gives two rough limits, and the lower one is the server's capacity:

• CPU: cores × (1000 ÷ ms of CPU per request). A medium server has 4 cores and an Inkwell request needs about 3 ms of CPU, so 4 × (1000 / 3) is about 1 300 to 1 400 requests per second.
• Threads: threads ÷ time each request holds one. This is Little's Law: requests in progress = arrival rate × time each one takes. With 50 threads and requests that take 20 ms, a server finishes at most 50 / 0.020 s = 2 500 requests per second.

For Inkwell today the CPU runs out first. Threads run out first when requests spend a long time waiting on the database: at 100 ms each, 50 threads cover only 500 requests per second. A thread waiting is a thread not serving anyone else. In retrystorm, a component's Metrics tab shows each of these limits filling up during a run.

Utilization and queues

Utilization is how much of a limit is in use. When requests arrive faster than they can be handled, they wait in a queue. Waiting grows slowly at first, then sharply as utilization nears 100%:

Past 100% the queue fills up and new requests are turned away as errors. That's why nobody runs servers at 99%: you keep headroom for bursts like a launch.

Scale up or scale out

• Scale up (vertical): a bigger machine with more cores. Simple, but machines only get so big, the biggest are expensive, and it's still one machine.
• Scale out (horizontal): more copies of the server, called replicas. Because Inkwell's app servers are stateless, any replica can serve any request, so you can keep adding them.

Statelessness is what makes this work. If a server kept a user's login session in its own memory, every request from that user would have to return to the same server (sticky sessions), and a crash would log them out. Keeping sessions in a shared store, or in a signed token the client sends, lets any replica serve anyone.

Load balancers

With several replicas, something must decide which one handles each request: a load balancer. It sits in front of the replicas and spreads requests across them.

Any replica can serve any request, so the load balancer spreads them out.

Common strategies are round robin (take turns), least connections (send to the least busy replica), random, and consistent hash (the same key always goes to the same replica). In retrystorm, an app server with more than one replica must have a load balancer in front of it.

Bottlenecks move

Fix one limit and traffic flows further, until it hits the next one.

Scaling the app layer sent more traffic to the database, which became the new limit.

A database accepts only so many connections, open conversations with app servers, at once; when all of them are busy, new queries have to wait, and in retrystorm they're refused. A struggling app layer can hide that limit: relieve the app servers, and the database starts refusing connections instead. Scaling is a loop: find the resource that's full, relieve it, measure again.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

Your practice

Survive the Launch: traffic quadruples for ten seconds. Scale Inkwell's app layer out behind a load balancer, then find what breaks next.