A system can be failing while every dashboard is green, and it can page someone at 3 am for a blip nobody noticed. Observability is choosing what to measure so that you notice real trouble early, and can tell from the numbers which part is the cause.
Averages hide the slow ones
The average is the number everyone reaches for, and it's the one that hides the most. A page that takes 10 ms for most readers and 400 ms for a few has a low average. The few still wait, and they're the ones who complain.
Latency is better described by percentiles. The p50, or median, is the time half the requests beat. The p99 is the time 99 out of 100 beat, so one request in a hundred is slower. At 1 000 requests a second, that's ten readers every second.
Three services below all average 40 ms. Pick each, and then change the sliders: the average stays where it is, and the percentiles don't.
Four signals to watch
For any service you can start from the same four numbers, known as the golden signals:
| Signal | The question | In Retrystorm |
|---|---|---|
| Latency | How long do requests take? | p50 and p99 |
| Traffic | How much is being asked of it? | requests a second |
| Errors | How many requests fail? | the error rate, and why they failed |
| Saturation | How full is it? | queue depth, busy threads, connections in use |
Saturation matters because it moves first. Queueing starts well before a server is full: in a single-server queue at 70% busy, a request already waits about 2.3 times its service time before it starts, and at 95% about 19 times. Queueing is latency.
Error budgets
The availability chapter set a target: an SLO such as 99.9% of requests succeeding over a month. The other side of that target is the error budget: the 0.1% that's allowed to fail. At 1 000 requests a second, a 30 day month has about 2.6 billion requests, so the budget is about 2.6 million failed requests, or 43 minutes of total outage.
A budget turns "is it down?" into "how fast are we spending it?". The spending speed is the burn rate: a burn rate of 1 uses the budget exactly over the month, and 10 empties it in three days. One incident can use a large share of a month's budget, and how fast it's noticed and contained decides how much.
Finding the cause
An alert says something is wrong. It doesn't say what. The error rate is a symptom, and the same symptom can have different causes that need different fixes. What separates them is the breakdown: which component refused the requests, and what it said.
Each component reports its own failures, and the reason is a clue.
| What the component says | The usual cause | The usual fix |
|---|---|---|
| Refused or crashed mid-request | A dead replica still gets traffic | Health checks, more replicas |
| Queue full | Not enough capacity, or one slow dependency holding threads | More replicas, a faster dependency |
| Database has no free connection | Queries in flight exceed the database's connection limit | Faster queries, or a bigger tier or more nodes |
| Timed out | A slow dependency | Fix it, or shorten the queue in front of it |
| Rate limited (429) | A limit doing its job | Check the limit is set right |
Fix the biggest cause first and run it again: the second cause is often hidden behind the first. And read the whole path, not just the component that complains: a full app queue can come from a slow database behind it.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Percentile latency | Prometheus histograms, CloudWatch p99 metrics, Datadog |
| The four golden signals | Google's Site Reliability Engineering book |
| Error budgets and burn-rate alerts | Google's SRE Workbook, SLO tooling in Grafana and Datadog |
| Tracing a request across services | OpenTelemetry, Jaeger, AWS X-Ray |
Your practice
The pager went off at the evening peak: about one request in four fails. Read what each component says, work out the causes, and fix each one for as little as you can.