Inkwell has a public API now, and partners build on it: writing tools, publishing plugins, backup services. Every partner's contract allows 1 000 requests a second. This morning one partner shipped a bug that calls the API in a loop, at more than twice its contract, and the API's two app servers can't take it. Every partner's requests are failing, including the ones that did nothing wrong.
What it does
- Read and write: partners read and save notes through the API.
Workload
3 200 rps for 10 seconds, 90% reads, from two clients:
- partner, a well-behaved partner: about 800 rps.
- buggy, the partner in a loop: about 2 400 rps.
How to build it
The app tier is fixed: two medium replicas behind a load balancer, together good for about 2 700 rps. You can't add servers, so decide who gets turned away.
- Connect both clients to the load balancer, then the app servers, then the database. Run it and look at each client's errors.
- Put a rate limiter in front of the load balancer. Choose its algorithm, its scope, and its limit.
- The well-behaved partner should never notice, and the buggy one should still get exactly what its contract allows.