Inkwell's servers are fast. For users in Singapore, it barely helps, and the reason is physics.

Distance costs time

Signals in fibre travel at about two thirds of the speed of light, and real routes are longer than straight lines. A round trip from Singapore to Inkwell's servers in Virginia takes around 220 ms before the servers do any work at all. No database, cache or code change can remove it.

Before any work happens, the request has to cross the planet and back.

CDNs

A CDN (content delivery network) runs edge locations in cities all over the world, close to users. Put it in front of Inkwell: requests go to the nearest edge first.

• If the edge has a copy of the picture, it answers right there: a hit, typically 10 to 20 ms away.
• If not, the request travels on to Inkwell's servers (the origin), pays the long trip, and the edge keeps the response for next time.

Hits are answered nearby; only misses make the long trip.

A CDN is a cache that lives near users rather than near the database. Everything from the caching chapter applies: hit rates, skew, staleness.

Each edge keeps its own copies. A picture cached in London is still a miss the first time someone in Singapore asks for it, so every edge pays its own first view, and a picture viewed now and then in each region misses far more often than one global cache would. Most CDNs offer tiered caching (an origin shield): an edge that misses asks one regional cache before the origin, so the origin sees one fetch instead of one per edge. It saves the origin load, not the distance.

TTL and purging

Each cached copy has a TTL; when it runs out, the next request goes back to the origin.

The usual way to change a file on a CDN is to give the new version a new URL: avatar-v2.jpg, or a name made from a hash of its bytes. Nothing at the old URL ever changes, so the edge can keep every copy for a year and never needs a purge. Scripts, styles and most pictures are served this way.

Some content can't change its URL: an API response, a page, a count, or Inkwell's profile pictures, which the apps ask for at one fixed URL per user. When that content changes, the app can ask the CDN to purge its copies, through the CDN's purge API, or keep serving the old one until the TTL runs out (ttl_only in retrystorm). A purge isn't instant everywhere: the edge the upload came through drops its copy at once, and the purge reaches the others later, from a fraction of a second to minutes depending on the CDN (about a second in retrystorm). Until then, those edges keep serving the old copy. A purge also can't clear the copies browsers keep under their own max-age, so purged content usually gets a short browser max-age.

That's a dial between freshness and speed. Purging keeps pictures fresh, but a picture that changes often keeps being dropped, and its views go to the origin. Serving until the TTL keeps the edge full, with pictures up to one TTL old.

Medians, tails and what a CDN can't fix

A CDN moves the median a lot: most requests become hits close to users. The tail (p95, p99) is a different story, because it's made of the requests the edge couldn't answer. Those still pay the full distance:

• Uploads (writes) always travel to the origin.
• Misses, including the very first view of each picture, and every view after a purge or an expired TTL.
• Content that is different for every user and can't be shared at all.

That last point decides what goes on a CDN. Static content, the same bytes for everyone (pictures, scripts, a public post), caches well. Dynamic content, built per user (a feed, a cart), mostly can't, unless it's split into a shared part and a small personal part. An edge keys each copy by the URL (its cache key), so two URLs for the same picture are cached twice, and a user id in the URL makes every copy private.

However fast the servers get, distance adds time that no code can remove.

Edge functions

A CDN answers a hit without running any of your code. That's why a hit is cheap and fast, and also why the origin never hears about it: whatever the origin does on each request (count it, check who's asking, pick an answer by country) doesn't happen for the requests the edge answered.

Edge functions close that gap. You deploy a small function to every edge location, and it runs on every request, in the city the request came from. It can answer from a cache at the edge just as a CDN does, and still do work per request: send an event about it, check a header, choose between two answers. Work it hands off, like sending an event, happens after the answer has gone back, so the user doesn't wait for it.

What it can't do is bring the origin closer. A function that needs data its edge doesn't hold still makes the long trip, and it adds its own run time, a millisecond or two, to every request. Each request also costs more than a plain CDN hit.

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

Your practice

Serve the World: bring Inkwell's profile pictures close to users on three continents, then tune the edge so almost everyone is fast without pictures going too stale.