Inkwell's profile pictures, viewed by users on three continents. Your servers live in one place, and the speed of light is not negotiable.

Inkwell's apps, including old versions on phones that never update, ask for each user's picture at one fixed URL. A new picture can't get a new address: it replaces the old one at the same URL.

What it does

• Read: someone views a profile picture.
• Write: a user uploads a new picture, at the same URL.

Workload

A steady 2 000 rps, 99% views / 1% uploads, for 20 seconds, over 1 000 pictures. Popular profiles get most of the views, and they change their picture the most too.

Global users

Every request to your servers pays that round trip, however fast the servers are.

What a CDN does

A CDN keeps copies of your content in edge locations close to users. Put it in front of everything (client → CDN → load balancer → ...): a picture it has cached is answered from the nearby edge, and anything else goes on to your servers. Each region's edge keeps its own copies, so a picture cached in us-east is still a miss the first time someone in ap-southeast asks for it.

How long the edge keeps a copy is its TTL. Its On write setting decides what an upload does:

• purge: drop the copies, so the next view fetches the new picture from your servers. The uploader's edge drops its copy right away; the purge reaches the other regions about a second later.
• ttl_only: keep serving the old copy until the TTL runs out. Views stay fast, but a new picture can take up to one TTL to appear.

Tiers

• Easy: stays up. Almost every request succeeds.
• Medium: fast for most. Median under 40 ms across all three regions. A cache next to your servers doesn't help here: the distance is the problem.
• Hard: fast for almost everyone. 95% of requests under 115 ms (the p95), and a new picture shows up within 10 seconds.

Hints

• Popular pictures change often. Think about what purging on every upload does to how often the edge has a copy.