Inkwell Pay moves a few cents from a reader to a writer: a tip. It is the opposite of the news feed. The volume is small, and a mistake is a person's money: a confirmed tip that's gone, a tip recorded twice, a ledger that stops adding up. This lesson follows the interview order, and the deep dives note which chapters they build on.
Ask before you draw
| Question | Why it matters | Inkwell Pay |
|---|---|---|
| What must never happen? | It decides what the design spends its effort on | A confirmed tip lost, a tip recorded twice |
| What is the source of truth? | Everything else must be rebuildable from it | The ledger |
| How fast does a tip need to be? | Sets how long a caller may wait | A couple of seconds, and a short pause during a failure is fine |
| What if the database fails? | Decides replication and failover | Keep taking tips, lose none |
| Can a tip be undone? | Decides whether history can be edited | Never edited: a refund is a new entry |
Write down the first row. "Never lost, never twice" is the sentence the design has to earn, and it's what separates this from most systems, where a little loss or repetition is acceptable.
Estimate the load
Work the chain yourself first, from the assumptions in the worksheet. The worked numbers open once you've done it.
API and data model
POST /tips takes the reader, the writer, an amount and an idempotency key the client makes up for this tip, so a retry of the same call is recognised. GET /receipts/{id} returns what was recorded.
- Ledger: append-only entries (account, amount, tip id, time). Every tip adds two entries that sum to zero, which is double-entry bookkeeping, and a total that doesn't add up is a bug you can detect.
- Balances: derived from the ledger, kept for speed, and rebuildable from it.
- Idempotency records: the key and the result of the first request with it.
The card charge is slow and can fail, so it's a separate step: a tip is recorded as pending, the charge happens, and the tip moves to captured. Record the tip before you ask the payment provider (such as Stripe) to charge the card, never after, so a crash in between leaves a pending tip to reconcile and not a charge nobody recorded. Pass the same idempotency key to the provider, so a retried charge is not a second charge. The card network itself takes no such key: the provider is what spots the repeat. And a card charge of a few cents costs more in fees than it moves, so real tipping systems charge a balance the reader tops up, or batch a reader's tips into one charge.
Don't lose a confirmed tip
A tip is confirmed only once it is on two machines.
"Confirmed" has to mean something. If the primary confirms a tip and copies it to the replica afterwards, a crash in between loses a tip the caller was told was safe. With synchronous replication the primary waits for the replica before it confirms, so a tip that was confirmed exists on both. With just one synchronous replica, though, that replica going down stalls every write, so production setups keep two and wait for either one (in PostgreSQL, ANY 1 (a, b)).
Retry safely through a failover
While the replica is being promoted, writes fail. The caller can give up, or retry until the database is back. Retrying has two risks: a retry of a tip that had already been recorded records it twice, and retries from every caller at once can swamp the new primary.
Deep dives
| They ask | What to say | Chapter |
|---|---|---|
| The payment provider times out | The tip stays pending. A reconciliation job later asks the provider what happened to that charge and settles it | Timeouts and retries |
| A reader double-clicks | The same idempotency key, so the second request returns the first result | Idempotency |
| A writer gets a thousand tips a second | A hot row. Keep per-tip entries, and update the balance in batches or in split counters | Hot keys |
| A refund | A new, reversing entry. Never edit or delete an old one | Durability |
| The ledger and the payment provider disagree | Reconcile nightly, find the difference, and fix it with entries | Consistency |
| Receipts are slow | Read from the replica, and route a reader to the primary right after their own tip | Consistency |
| A tip goes through several states | A state machine: created, pending at the provider, then succeeded or failed, and refunded only from succeeded. Each move is a conditional update from the state it expects (SET state = 'succeeded' WHERE id = ? AND state = 'pending'), so a late or repeated event can't move a tip backwards | Transactions |
| The provider reports the outcome later, by webhook | Check the signature, record the event id with an insert that fails on a repeat, answer at once and do the work from a queue. Webhooks arrive late, twice or out of order, so apply them through the state machine, and let the nightly reconciliation catch any that never arrive | Idempotency, Queues and workers |
Choose two, and say which comes first and why.
Say the tradeoffs
| Choice | One side | The other side |
|---|---|---|
| Replication | Synchronous: nothing lost, a little slower, stalls if the replica is down | Asynchronous: faster, loses confirmed tips on a crash |
| Failover | Automatic: seconds of downtime | Manual: slower, but a person decides, so no risk of two primaries |
| Retries | Patient: tips survive a failure, callers wait | Impatient: quick failure, tips dropped |
| Storage | One database: simple, consistent totals | Sharded: scales further, harder to keep consistent |
Commit to one relational database with two replicas, synchronous replication and automatic failover, an idempotency key on every tip, and callers that retry until the failover completes. Then say what you'd watch: lost or duplicate tips (both must be zero), failover time, and the difference between the ledger and the payment provider.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Idempotency keys on a payments API | The Idempotency-Key header in Stripe's API |
| Synchronous replication | PostgreSQL synchronous_commit, Amazon RDS Multi-AZ |
| Automatic failover | Patroni for PostgreSQL, Aurora's failover to a replica |
| Append-only, double-entry ledgers | Accounting systems, and the ledger services payments companies describe |
Your practice
Inkwell Pay lost some confirmed tips, recorded some twice, and showed errors for ten seconds when the primary crashed. Fix all three, and then make the wait for a tip as short as you can.