Design Inkwell Notify, the service that tells readers things: password resets and login codes, new followers, weekly digests. Two kinds matter:
- Urgent: a reset code that arrives a minute late is useless. It must go out within 1.5 seconds.
- Bulk: digests and follower alerts. They can wait, but each one has to go out, once.
The email and push providers are slow, about 100 ms a send, and flaky: about 3% of sends fail. Support is hearing three complaints: some readers never got their reset email, some got a digest twice, and when the digests go out, the resets are late.
What it does
- Write: an event happens (a reset is requested, a digest is due), and the service sends the notification.
Workload
A steady 500 events a second for 15 seconds: 40 urgent a second from urgent (keys 0 to 999) and 460 bulk a second from bulk (keys from 1 000 up).
What's fixed
The app servers, and the providers' speed and failure rate, are fixed.
How to build it
A queue and a pool of workers for each kind are already on the canvas: urgent-queue with urgent-send, and bulk-queue with bulk-send. As found, each queue gives a job one attempt and offers it again after 80 ms if the worker hasn't answered. You choose each queue's settings and each pool's tier, size and idempotency.
Steps
- Build the path: both clients, load balancer, app server. Then, on the app server's connections, send urgent events (keys below 1 000) to the urgent queue and bulk events (keys from 1 000) to the bulk queue. Connect each queue to its workers.
- Run it as it is. Look at the jobs dead-lettered and done twice.
- Fix both, then give each pool only what its own deadline needs.