Design Inkwell Drive, where writers share big files with their readers: drafts, images, whole manuscripts. Files run from a few megabytes to tens of megabytes, readers are on three continents, and the first version routes every byte through the app servers and the database. Last week the database's network card filled and the whole service fell over.
What it does
- Folders: users list folders and change sharing settings, small requests of about 2 KB through the app servers and the database.
- Downloads: readers open shared files, about 8 MB each, a few far more popular than the rest.
- Uploads: writers upload new files, about 40 MB each.
Workload
A steady 1 000 rps for 15 seconds: 600 rps of folders, 380 rps of downloads from readers in three regions, and 20 rps of uploads. There are 20 000 files.
| Region | Share | Round trip to your servers | Round trip to the nearest edge |
|---|---|---|---|
| us-east | 40% | ~10 ms | ~8 ms |
| eu-west | 30% | ~75 ms | ~14 ms |
| ap-southeast | 30% | ~220 ms | ~18 ms |
What's fixed
The app servers are fixed, and folders go through them and the database. Every upload first asks the app servers, through the load balancer, for permission to send its file.
Uploads here travel at datacenter speed. A real user's connection is far slower, so a real 40 MB upload takes seconds, and one sent through the app servers holds one of their connections for all of it.
How to build it
Files can go anywhere that stores them: the database, object storage, and a CDN in front of either. Uploads need a presigned URL from the app servers: connect uploaders to the load balancer as well as to storage.
Steps
- Build the path for folders: client, load balancer, app server, database. Then send the files the same way, and look at the database's network.
- Give the files a home of their own, and look at the downloads and the uploads.
- Then make it cost as little as you can.