Inkwell Notify tells readers things: password resets, new followers, the weekly digest. It looks like "send an email", and it's hard because there are two kinds of message with different deadlines, the providers are slow and flaky, and a mistake is visible to a real person: a message that never arrives, or one that arrives twice. The lesson follows the interview order, and the deep dives point back to the chapters they use.
Ask before you draw
| Question | Why it matters | Inkwell Notify |
|---|---|---|
| What kinds of notification? | Decides whether one deadline fits all | Urgent (resets, login codes) and bulk (digests, follower alerts) |
| How late is too late? | Sets the latency budget per kind | Urgent within 1.5 seconds, bulk within half a minute |
| Which channels? | Each provider has its own speed and failures | Email and push to start |
| What if a send fails? | Retrying risks duplicates, giving up risks loss | Never lost, effectively sent once |
| What if a provider is down? | Decides what must be stored and for how long | Keep trying, don't lose the backlog |
Write the answers down. The rest of the interview comes back to them, and "never lost, effectively sent once" is the sentence the design has to earn.
Estimate the load
Work the chain yourself first, from the assumptions in the worksheet. The worked numbers open once you've done it.
The simulator runs a much smaller version with the same shape: 500 events a second, 40 of them urgent.
API and data model
One call takes an event: POST /events with the user, the kind of notification and an idempotency key, so a caller that retries the call doesn't send twice. Everything else happens behind it.
- Events: the record of what was asked, one row per event, kept so a lost send can be found and replayed.
- Preferences: per user, which channels they want and when not to be disturbed. Read by the workers before they send.
- Delivery log: per event, whether it was sent, to whom, and when. It's what answers "did I get it?"
The queue is the buffer between the call and the send. An event is acknowledged once it's safely queued, not once it's sent, so a slow provider never slows the caller.
One queue or two
The design is a pipeline: the app takes the event and puts it on a queue, workers take jobs and call the provider.
Two queues, so a digest can never make a reset wait.
Send everything through one queue and the digests, which are most of the volume, fill it. A reset that arrives behind a backlog of digests waits as long as the backlog does.
When a send fails or runs slow
About 3% of sends fail, and the queue decides what happens to those jobs. One attempt per job means each failed send is dead-lettered: the user never gets it. More attempts, with a backoff between them, mean the failure is retried and almost always succeeds.
The second setting is the visibility timeout: how long the queue waits for a worker before it offers the job to another. If it's shorter than a send takes, the queue hands the job out again while the first send is still running, and the user gets the notification twice.
Making the workers idempotent hides most of the duplicates. Most providers, such as FCM, APNs and SES, don't spot a repeat for you, so the check is yours, and its order matters. Looking the notification id up in a delivery log, sending, and writing the log afterwards is a race: with a short timeout, two workers can both find the log empty and both send. Claim the id first instead. A conditional insert, on a unique key of the notification id, marks it as being sent, with an expiry in case the worker dies, and only the worker whose insert succeeds sends. A worker on a long send can also extend the job's visibility timeout. An APNs collapse id only replaces a notification that is still pending, so it isn't a check either. A worker that crashes after the send but before it marks the claim done can still repeat one once the claim expires, and the provider can't drop the copy, so delivery is still at least once, with duplicates made rare. A repeat still costs work too, so the timeout still has to fit the job.
Deep dives
| They ask | What to say | Chapter |
|---|---|---|
| The email provider is down for a minute | Keep retrying with backoff, and let the queue hold the backlog. Add a second provider if the outage is long | Timeouts and retries |
| A user gets a notification twice | The visibility timeout was shorter than the send, or a retry wasn't checked: claim the notification id with a conditional insert before sending, so only one worker sends it | Idempotency |
| Digests are delaying resets | Separate queues, and separate workers sized for each | Queues and workers |
| One user is flooded | A per-user rate limit, and batching events into one message | Rate limiting |
| The backlog keeps growing | Watch the age of the oldest job per queue, add workers, drop expired jobs | Pub/sub, Observability |
| A reset is useless after a minute | A message TTL: drop it rather than send it late | Pub/sub |
| A writer with 10 million followers posts | One job can't send 10 million. It fans out: a job pages through the followers and queues batches of about 1 000, and workers send each batch. Cap what goes to the providers, and keep urgent sends on their own queue so the burst doesn't delay a reset | Fan-out, Queues and workers |
| A digest goes out at 9 am in each reader's time zone | A scheduler: store each send with its due time, indexed by it. A job claims what's due in small batches, so two schedulers never take the same rows (a lease, or FOR UPDATE SKIP LOCKED in PostgreSQL), and puts it on the queue. A claim can repeat after a crash, so the send stays idempotent | Queues and workers, Idempotency |
Pick two and say which you'd cover first and why.
Say the tradeoffs
| Choice | One side | The other side |
|---|---|---|
| Attempts per job | One: simple, loses failed sends | Several: nothing lost, needs a duplicate story |
| Queues | One: cheapest, resets wait behind digests | One per kind: each kind has its own deadline |
| Duplicates | Idempotent workers: rare repeats, still costs the work | A longer timeout: no repeats from slow sends, but a crash before the ack still repeats, and failures are noticed later |
| Worker pool size | Sized for the peak: fast, and mostly idle | Sized for the deadline: cheaper, lags under load |
Commit to separate queues, three attempts with a backoff, a visibility timeout longer than the slowest send, and each pool sized for its own deadline. Then say what you'd watch: dead-lettered jobs, duplicates, and the age of the oldest job in each queue.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Queues with a visibility timeout and a dead-letter queue | Amazon SQS and its redrive policy |
| Mobile push through a provider | Apple's APNs and Firebase Cloud Messaging |
| Email and SMS providers | SendGrid, Amazon SES, Twilio |
| Idempotency keys on an events API | The Idempotency-Key header in Stripe's API |
Your practice
Inkwell Notify is losing some resets, sending some digests twice, and delaying resets when the digests go out. Fix all three, and then give each queue's workers only what its own deadline needs.