Machines fail: disks die, processes crash, deploys go wrong, a whole datacenter loses power. A system is available when it keeps serving through that, and the way to get there is to have a spare for everything and to notice quickly when something is gone.
Everything fails
One server might run for years without a problem. A thousand servers will lose one most weeks. At scale, failure isn't an exception to plan around; something is always broken somewhere.
Availability is the share of time (or of requests) a system serves correctly. It's usually counted in nines:
| Availability | Down per year | Down per month |
|---|---|---|
| 99% (two nines) | 3.7 days | 7.2 hours |
| 99.9% (three nines) | 8.8 hours | 43 minutes |
| 99.99% (four nines) | 53 minutes | 4.3 minutes |
| 99.999% (five nines) | 5 minutes | 26 seconds |
Teams pick a target per service. The SLI is what they measure (say, the share of requests that succeed), the SLO is the goal for it (99.9% a month), and an SLA is a promise to customers with a penalty attached. Each extra nine costs far more than the last.
Single points of failure
A single point of failure is any part whose loss takes the whole system down. One app server is one; so is one database, one load balancer, or one datacenter. The fix is redundancy: at least two of each, so one can fail while the other serves.
Health checks
A load balancer learns that a replica is dead with health checks: every few seconds it asks each replica "are you well?" (often a request to a /health path). After a number of failed checks in a row, it stops sending traffic there; after a few good ones, it sends traffic again.
Faster checks notice sooner but cost probe traffic, and with a threshold of one, a single slow answer takes a healthy replica out (flapping). Load balancers also watch real traffic (passive checks): a replica that keeps refusing requests is taken out without waiting for the next probe.
Database failover
A read replica can take over when the primary dies. Failover does it in three steps: notice the primary is gone, promote a replica to be the new primary, and point writes at it. Done automatically, that takes seconds to tens of seconds; done by a person woken at 3 am, much longer.
Two dangers come with it. If the old primary was only unreachable, not dead, two nodes may both believe they're the primary and accept writes (split brain), so systems fence the old one (cut it off) before promoting. And a replica that was behind, usually by milliseconds but sometimes by seconds, takes over without those writes, which the durability chapter picks up.
In the lab, a monitor (a separate process that watches the primary's heartbeats and runs failover) loses touch with the primary when the network between them is cut, and it promotes the replica. Give the old primary a lease: the right to take writes for a fixed time, renewed while it stays in touch with the monitor. Cut off, it stops taking writes when the lease runs out. Compare a short lease with a long one.
Availability adds up
Parts a request needs one after the other multiply their availability: each one is another way to fail. Spares in parallel multiply their downtime: both must fail together.
| Design | Availability |
|---|---|
| One part at 99.9% | 99.9% |
| Two parts in a row, each 99.9% | 99.9% × 99.9% = 99.8% |
| Two spares side by side, each 99.9% | 1 - (0.001 × 0.001) = 99.9999% |
That's why every tier gets a spare, and why a long chain of services that each "has three nines" can be well under three nines together. The spare math assumes the two fail independently, though: a shared deploy, rack or region takes both down at once, and the seconds it takes to notice a failure and switch over count as downtime too. The same thinking applies one level up: a whole region can fail, so the most available systems run in several, either active-passive (a standby region takes over) or active-active (every region serves all the time).
Writing in more than one region
Active-active raises a question active-passive never asks: what if two regions change the same record at once? With one primary, every write has one place to go and one order. With a leader in each region (multi-leader replication), each region accepts writes locally and copies them to the others a moment later, so two regions can take conflicting writes to the same key before either hears of the other. Something has to settle it.
| Way to settle it | What happens | The cost |
|---|---|---|
| Last writer wins | Each write carries a timestamp, and the newest one stays | The other write is lost with no error, and clocks that disagree pick the winner |
| Merge | Writes that combine are combined: two likes add up, two items added to a cart both stay (data types built for this are called CRDTs) | Only data with a sensible merge; an edited title has none |
| A home region per record | Each key's writes go to one region, and the others forward them | Far-away writers pay the distance, and a region outage needs a failover for its keys |
| Keep both, ask later | Both versions are kept, and the app or the user picks. Each version carries a version vector, a counter per region of the writes it has seen; when neither vector is ahead of the other, the versions conflict | Every reader has to handle two answers |
Most designs avoid conflicts rather than settle them: a user's data is written in the user's home region, and only data that merges, such as counters, is written everywhere. In an interview, say which you'd do for each kind of data, and what happens to a write when two regions disagree.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Health checks | AWS load balancer target health checks, Kubernetes liveness and readiness probes, Envoy outlier detection |
| Database failover | Amazon RDS Multi-AZ, Aurora (failover in about 30 seconds), Patroni for PostgreSQL |
| Leader election and fencing | etcd and ZooKeeper, which Patroni and older Kafka use |
| Multi-region writes | DynamoDB global tables (last writer wins in the default mode), Cassandra across data centres (leaderless, rather than multi-leader, with the same conflicts), CRDTs in Riak and Redis Enterprise |
| SLOs | Google's Site Reliability Engineering book |
Your practice
Last night an app server died, and a third of Inkwell Blogs' requests failed for as long as it was down. Then the database's primary went down, and nobody could publish until someone woke up and fixed it. Make Blogs ride out both on its own.