Retrying is how a system rides out a slow moment, and it's safe for reads. For a write it can do the work twice, and some work, like charging a card, must not happen twice. This chapter is about making a retry harmless.

A retry can run the work twice

A timeout only tells the caller to stop waiting. It doesn't cancel the work: the server carries on, and the save may finish a moment after the caller has gone. The caller doesn't know that, sees a failure, and sends the request again. Now the same write has been applied twice.

The first save finished at 400 ms. Nobody told the retry.

A request is idempotent when running it twice leaves the same result as running it once. "Set the title to Draft 2" is. "Add a like" and "charge 5 dollars" aren't: each run changes the result. Only requests like those need protecting.

Idempotency keys

An idempotency key is a unique id the client makes up for one operation, and sends on the first try and on every retry of it. The server keeps a record of the keys it has handled and their results.

The keys live in the same database as the work. A repeat finds K, gets the stored result, and the work isn't done again.

A few details decide whether it works:

• The client makes the key, once per operation. A fresh key for each retry would defeat the point.
• Check and reserve atomically. If two copies of a request arrive at the same moment, only one may start the work. Store the key in the same transaction as the work (a group of writes the database saves together or not at all), or with a unique constraint (a rule that refuses a second row with the same key).
• Keep keys long enough. The store must remember a key for longer than any retry can arrive. Stripe keeps them for 24 hours. If retries can arrive up to 30 seconds later and 200 writes a second carry keys, that's about 6 000 keys.
• A key stands for one request. The same key with a different body is a bug, and should be refused rather than answered with the first result.

Try it on a tip that takes longer to save than the app will wait. Switch the key on first, and watch what the app does with the retry.

Shorter timeouts, more repeats

The shorter the timeout, the more callers give up on saves that were about to finish, and the more repeats there are.

Make the operation idempotent itself

Sometimes there's no need for a key store, because the operation can be written so that repeating it changes nothing.

The last one is a conditional write: the second try finds the version has moved and does nothing. Setting a value, using a client-made id and checking a version all turn "do it again" into "already done". Where that's possible it's simpler than keeping keys.

Setting a value has a catch. If another tip lands between reading 30 and writing 40, the set wipes it out. For a balance, pair the set with a version check (the last row), or save each tip under its own id (the second row) and add them up.

Exactly once

A network can lose a message, and a sender that can't tell loss from slowness has to send again. So messages arrive at least once, sometimes twice. "Exactly once" isn't something the wire gives you. It's the effect you get by combining at-least-once delivery with a receiver that ignores repeats, using a key or an idempotent operation.

The key has to travel as far as the effect does. If your service charges a card through a payment provider, pass the same key on to the provider, or a retry of your own request will still charge twice behind your back. Queues have the same shape: a job may be delivered twice, so the work it does must be safe to repeat.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

Your practice

Readers can tip writers, and some were charged twice for one tip. Find the setting that makes a retry safe, then keep failures and waiting low.