A page is rarely one call, and it's only as fast as the slowest call it waits for. Budgeting each call's time is how big systems stay fast.
A page is many calls
To build its home page, Inkwell's app server needs the user's notes, their profile, today's prompt and some recommendations. It asks for all of them at once, in parallel, and builds the page when the answers are back.
Four calls at once. The page waits for the slowest.
Recommendations come from another team's key-value store, a database that only looks values up by a single key. A later chapter covers how it works; here it's just one more call.
In parallel, the page takes as long as its slowest call, not the sum. That's much better than calling them one after another, but it has a catch.
The tail grows with every call
Suppose each service is slow 1% of the time. A page that calls one of them is slow 1% of the time too. A page that calls six is slow whenever any of the six is. It's fast only if all six are fast, so with n calls each slow with chance p, the share of slow pages is 1 − (1 − p)^n. For six calls that's 1 − 0.99^6, about 5.9%. The page's p99 is set by much rarer slow calls than any single service's p99.
| Calls per page | 1 | 6 | 100 |
|---|---|---|---|
| Pages that hit a 1% slow call | 1% | 6% | 63% |
Each call below is fast, except that 1% of them take ten times as long. Change how many calls the page makes and watch the share of slow pages follow the formula.
A budget for every call
A latency budget starts from the target for the whole page, say 99% of feed pages within 200 ms, and gives each call room inside it. Because the calls run in parallel, each one can use most of the budget. Each call then gets a deadline: if it hasn't answered by then, the page stops waiting for it.
| Feed page, 200 ms budget | Deadline | If it misses |
|---|---|---|
| Posts | 160 ms | the page fails |
| Profile | 160 ms | the page fails |
| Who to follow | 140 ms | the box is left out |
A deadline on a call the page can't do without just trades slowness for failure. The page waits less, then fails anyway. That's only right when a late answer is as useless as none.
Optional calls
Some parts of a page are nice to have. Without its "who to follow" box, the feed page still works: it shows the posts and leaves that box out. Marking the call optional with a deadline means a slow suggestion can't hold the page up or break it; the page simply goes without.
This is called graceful degradation: under trouble, show less rather than nothing.
Choosing the deadline
An optional call's deadline is a trade. Shorter, and pages are faster but miss the extra part more often. Longer, and the part shows up more often but slow calls push the page toward its budget.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Tail latency under fan-out | Google's paper "The Tail at Scale" |
| Deadlines and optional calls | gRPC deadlines, fallbacks in Netflix's services |
| Hedged requests | sending a second copy of a slow call and taking the first answer, also from "The Tail at Scale" |
Your practice
Inkwell's new home page shows a writer's notes and a box of recommendations from a store that's usually quick but sometimes very slow. Since it shipped, the slowest pages have taken a quarter of a second. Get 99% of pages under 100 ms while still showing recommendations on almost every page.