Rate limiter: what it does in a system design

Home › Reference › Components › Rate limiter

Rate limiter

Lets through at most a set number of requests per second, overall or per client, and turns the rest away at once with a 429.

Algorithm: How the limit is counted. Token bucket allows short bursts then a steady rate. Fixed window counts per clock window and can let double through at a boundary. Sliding window never exceeds the limit in any window.
Limit applies to: Whether one limit is shared by everyone or each client gets its own. Per client stops one noisy client from using up the allowance of the others.
Limit (requests/s): How many requests per second are let through. Requests above it are refused at once with a 429 (too many requests), which keeps load off what is behind it.
Burst (bucket size): Token bucket only. How many requests can pass back to back after a quiet spell. Bigger absorbs spikes better but lets more load through at once.
Window (ms): Fixed and sliding window only. The length of the counting window. The allowance per window is the limit times this length, so it must hold at least one request.
Decision latency: Time the limiter takes to decide, added to every request it sees. Pick the shape: constant or lognormal (mostly fast, some slow).