Inkwell Shop sells prints of writers' work. A database is careful: it keeps every product on disk and answers every question correctly. That care costs milliseconds, thousands of times a second. A cache trades some of it away for speed.

Why caches work: skew

Real traffic is rarely spread evenly. In Inkwell Shop, a handful of bestselling prints get most of the views, and most products get almost none. This pattern has a name, a Zipfian distribution, and it's everywhere: posts, links, videos, songs.

That's good news: keep the popular items in memory and you can answer most requests without touching the database.

Hits, misses and hit rate

• A hit is answered from the cache, in a fraction of a millisecond.
• A miss goes on to the database; the answer is then kept in the cache for next time.
• The hit rate is the share of requests that hit.

Average cost is roughly hit rate × hit time + miss rate × miss time:

A cache also protects the database: hits never reach it, so the database only sees the misses and the writes. It has a ceiling of its own, though: one cache node answers somewhere around 25 000 to 200 000 requests a second, depending on its size, and past that, requests line up for it like for any busy server.

Where it sits

There are two common layouts. In cache-aside, the app talks to both: it checks the cache, and on a miss it reads the database itself and puts the answer in the cache. In a read-through cache, the cache sits inline between the app and the database: the app only asks the cache, and the cache fetches its own misses. Retrystorm's cache sits inline like that. Reads go through it, writes pass through it on their way to the database, and its write strategy decides what happens to the cached copy. Either way, every hit is a query the database never sees, and the database only hears about the misses and the writes.

The hard part: writes

When a writer changes a print's price, the cached copy is suddenly wrong. Serving it is a stale read. What the cache does on a write is its write strategy:

• invalidate: drop the cached copy; the next read fetches the new price.
• write-through: replace the cached copy with the new price.
• bypass: the write goes on to the database and leaves the cached copy alone (write-around, when nothing deletes the cached key either). It stays wrong until it's evicted (pushed out to make room), or until a TTL (time to live) expires it.

A fourth strategy, write-back (or write-behind), writes only to the cache and saves to the database later, in batches. Writes get very fast, but a cache that crashes before saving loses them, so it suits data like view counters, not prices or orders.

With invalidate, the write clears the cached price on its way to the database.

Here the price changes at 100 ms and a reader asks for it at 0, 200, 400 and 800 ms:

Even with invalidation there is a tiny window: a read that was already on its way to the database when the price changed can come back with the old value. Some caches guard against it with a lease: the cache refuses to store an answer that was fetched before the latest invalidation, so stale reads last milliseconds, not minutes. The idea goes back to Gray and Cheriton's leases (1989), and Facebook's memcache made it the standard fix for this race. Memcached's meta commands (version 1.6 and later) give you the pieces; stock Redis doesn't, so with Redis the app builds it or leans on a short expiry.

When a cache doesn't help

A cache only pays off when the same data is read again before it changes. Two situations break that:

• Even traffic: if every product were equally popular, a cache holding 5% of them would hit about 5% of the time. Most reads would still reach the database, plus the cache's own small overhead.
• Write-heavy data: if items change constantly, cached copies keep getting dropped. Picture the like count on a viral post, changing hundreds of times a second: by the time it's cached, it's already out of date.

Before adding a cache, ask two questions:

1. Is the traffic skewed? A few hot items should get most of the reads.
2. Do those items change slowly compared to how often they're read?

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

Your practice

Cache the Hot Pages: make Inkwell Shop's pages fast with a cache, without ever showing old prices.