Retries and backoff help one caller ride out a short failure. They do nothing for a service that is failing for everyone at once: every caller keeps knocking, and the knocking is part of why it stays down. In this chapter you add a part that notices when calls are failing and stops making them.
A failure that feeds itself
After the 8:00 rush, Inkwell's servers don't recover. Each replica holds up to 2 000 requests in its queue, and callers give up after 300 ms and retry. A request that waits behind a long queue is answered after its caller has left, so the work is wasted, and the retry joins the back of the same queue. The load never falls, even though the rush is over.
A circuit breaker sits between the caller and the service. It watches how calls end. When enough of them fail, it opens and refuses new requests at once, without calling the service, so the queues can drain.
Closed, open, half-open
A breaker is a small state machine. It starts closed, passing every call through and keeping a tally of recent answers. When the share of failures in the last few seconds passes a threshold, and enough calls were made for the share to mean something, it trips.
| State | What the breaker does | Moves on when |
|---|---|---|
| Closed | passes calls through and counts failures | the failure rate passes the threshold: open |
| Open | refuses every call at once | the cooldown ends: half-open |
| Half-open | lets a few probe calls through, refuses the rest | all probes succeed: closed; one fails: open again |
When the breaker is open, the request never reaches the servers.
A slow answer can count as a failure too. In a storm most calls don't fail outright; they come back after the caller stopped waiting, and a breaker that only counts errors notices late.
Choosing the settings
Every setting is a trade between protecting the service and refusing requests that would have worked. A low failure threshold and a small number of required calls trip early, and a long cooldown keeps the breaker open for longer. The cooldown matters most: too short and the breaker probes a service that is still struggling, too long and callers are refused long after the service is healthy.
There is no cooldown that is right everywhere. It should be a little longer than the service needs to recover, and the half-open probes are how the breaker finds out whether it has.
What a breaker can't do
A breaker only knows how its own calls ended. It protects one dependency, so each one needs its own, and it turns a slow failure into a fast one: callers still get an error. What they do with that error is a separate decision, such as showing a cached page or leaving out an optional part of the screen. The breaker also doesn't fix the cause. Here the cause is queues far longer than any caller will wait, and shortening them is the real cure.
In the wild
| Idea | Where you'll meet it |
|---|---|
| Closed, open, half-open | resilience4j and Polly, the usual libraries for it |
| Tripping on a failure rate over a window | Hystrix, which made the pattern common |
| Ejecting failing hosts | Envoy's outlier detection |
Your practice
After the morning rush the servers' queues keep callers retrying into a backlog, and most requests fail until the queues drain. You can't shorten the queues or add servers. Add a circuit breaker in front of them and tune it so the queues drain quickly and few requests fail. Medium asks for fewer than 38% of requests to fail and 99% to take under a second.