Every request can fail or take too long. What the caller does next (wait, give up or try again) can turn a short rush into a small blip or into an outage that outlasts the rush itself.
Nobody waits forever
A timeout is how long a caller waits for an answer before giving up. Without one, a caller stuck on a slow server waits indefinitely, holding a connection and a user's patience the whole time. With one, the caller gets a clear failure it can act on: show an error, use a fallback, or try again.
The app stops waiting. The server doesn't know, and keeps working.
There's a catch that matters later in this chapter: giving up doesn't cancel anything. The request is still sitting in a server's queue, and the server will still spend its time on it, even though nobody is waiting for the answer any more.
See it happen: the app gives up after 150 ms, and the server needs 600 ms in all. Tell the server when the app gives up, then try delaying that message.
Retries hide brief failures
Many failures are brief. During a short rush, a server with a full queue turns requests away; a moment later it has room again. A retry sends the failed request again, and if it arrives after the rush, it succeeds and the user never notices.
The timing is everything. A retry sent immediately arrives in the same rush that rejected the first try, and gets rejected the same way. A retry sent after the rush ends, a few seconds later here, gets through.
Backoff and jitter
Exponential backoff waits longer before each new try: for example 1 s, then 2 s, then 4 s, up to a limit. A server that is struggling gets more and more room to recover.
Jitter makes each wait a random time up to that limit, so a thousand callers that failed at the same moment don't all come back at the same moment too.
| Try | Backoff | With jitter, waits |
|---|---|---|
| 1st | none | right away |
| 2nd | 1 s | anywhere from 0 to 1 s |
| 3rd | 2 s | anywhere from 0 to 2 s |
| 4th | 4 s | anywhere from 0 to 4 s |
Always cap the number of tries. Every retry is extra work for a server that is already short of capacity; a caller that retries forever never lets it recover.
Retrying writes safely
Retrying a read is harmless. Retrying a write may not be: if the first try timed out after the server saved it, the retry saves it again. Two posts, two orders, two payments.
A request is idempotent when sending it twice leaves the same result as sending it once ("set the title to X" is; "charge $5" isn't). Make non-idempotent writes safe to retry with an idempotency key: a unique id the client sends with the request, so the server can return the first answer to a repeat instead of doing the work again. The Idempotency chapter covers how it works.
| Request | Safe to retry as is? |
|---|---|
| Open a note | yes |
| Set a note's title to "Draft 2" | yes |
| Add a like, place an order, charge a card | only with an idempotency key, or made idempotent another way |
Retry storms
Now put the pieces together badly. Servers let requests wait in long queues; callers give up after 300 ms and retry. During a rush, requests wait longer than 300 ms, so callers give up and send them again. The servers now hold the original request (which nobody wants any more) and the retry. The queue grows, more requests time out, more retries arrive.
That loop is a retry storm. It can keep a system down long after the rush that started it has ended.
Circuit breakers
Backoff spaces out one caller's retries. A circuit breaker stops a caller from calling a service that is clearly failing at all. It counts recent failures and moves between three states:
| State | What the caller does | Moves on when |
|---|---|---|
| Closed | calls as normal | failures pass a threshold: opens |
| Open | fails at once, or uses a fallback, without calling | a cool-down passes: half-open |
| Half-open | lets a few trial calls through | they succeed: closed; they fail: open again |
An open breaker gives the struggling service room to recover and gives users a fast error or a fallback instead of a slow timeout. The cost is that some requests fail that might have worked; the threshold and cool-down decide how many. The chapter on circuit breakers builds one and tunes it.
Timeouts that fit the system
A timeout should be longer than a normal answer takes, including the waiting a busy moment adds, and short enough that users aren't left staring at a spinner. Too short, and callers abandon work the servers are about to finish.
The other half of the fix is on the servers: keep queues short, so an overloaded server turns requests away in a millisecond instead of letting them wait past every caller's timeout. A fast "no" is something a caller can retry later; a slow one is just wasted work.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Backoff with jitter | the default retry policy in the AWS SDKs |
| Timeouts passed from call to call | gRPC deadlines |
| Idempotency keys | the Idempotency-Key header in Stripe's API |
| Circuit breakers | resilience4j, Envoy's outlier detection, Netflix Hystrix before it |
Your practice
Every weekday at 8:00 Inkwell sends a writing prompt, and a few seconds later a wave of users opens the app at once. The app gives up after 300 ms and retries right away, and the morning rush ends with an error for one request in eight. You can't add servers. Change the app's retry policy so it rides out the rush.