retrystorm · Learn · Interview practice · Glossary · Components

Home › Learn › URL Shortener

Chapter 36 · Case studies · 13 min

URL Shortener

Design and build ink.ly end to end: requirements and numbers, unique codes, fresh reads after an edit, redirects fast worldwide, and what happens when each box fails.

Codes are ready, and now the whole of ink.ly has to be built: mint a unique code for every link, never lose one, redirect fast worldwide, and handle edits correctly. The founders remember your whiteboard sketch, and want to hear how the finished design closes the gaps you named.

Back in The Design Interview you sketched ink.ly on a whiteboard and said where the sketch would break: one database can't serve 12 000 redirects a second, and readers on other continents pay the distance on every click. Now you have the boxes for both. Run it the way an interview would: requirements and numbers, the design, then the deep dives the interviewer will push on.

Requirements and numbers

The requirements haven't changed since the whiteboard:

ink.ly
Functionalcreate a short link; redirect a short link; let the owner change where it points
Scale100 million new links a month; about 100 redirects per link created
Latencyredirects feel instant worldwide: tens of milliseconds
Consistencyafter an edit, the owner lands on the new target; everyone else within a few seconds
Durabilitya link that was created is never lost
Availabilityredirects keep working when a machine fails

Nor have the numbers: about 40 new links a second, 4 000 redirects a second on average and 12 000 at the evening peak, about 3 TB over five years. Clicks are skewed, a few links getting most of them, and readers are spread over three continents. The writes are easy, the reads are many and lopsided, and the distance is the latency problem.

High-level design

Draw the main paths, reads and writes, then justify each box from the requirements:

Each box is there because a requirement or a number asked for it.

  • The CDN is there for latency on three continents: a redirect it holds is answered nearby.
  • The cache is there for what the edges miss: each edge's first click on a link, every click after a purge, and links too new or too rarely clicked for an edge to hold. A few links get most clicks, so a small cache keeps those misses off the database.
  • The ID generator is there because many servers mint codes at once.
  • Synchronous commit on the database, not drawn as a box, is there because a created link may never be lost.

Compare it with the whiteboard sketch: every box added since answers one of the gaps you named then. That comparison is worth saying out loud in an interview.

Unique codes

Every new link needs a code no other link has, minted on many servers at once. The options from Unique IDs all work here, with different costs:

OptionGoodCosts
Counter handing out rangesshortest codes, one call per rangea central service; codes are guessable in order
Snowflake-style IDsno coordination per ID; each server needs its own machine numberlonger codes, about 11 characters
Hash of the URLno central servicecollisions to check and a retry when one hits; two owners of one URL share its code unless the owner is hashed in too

Code length is arithmetic. 62 characters (a to z, A to Z, 0 to 9) over 7 places give 62^7, about 3.5 trillion codes, against 6 billion links in five years: plenty, and short. Say why you'd pick ranges: the shortest codes, at the price of one more service to keep up.

Edits and fresh reads

The owner must land on the new target straight away; everyone else may see the old one for a few seconds. Every copy in the design can serve the old target: the cache, the CDN edge, a replica that hasn't caught up.

The edit path deals with each. It invalidates the link in the cache, so the next read fetches the new target. It purges the link at the edge, or the edge keeps it only for a short TTL, so other readers catch up within seconds. And the owner's reads right after an edit go somewhere that already has it, such as the primary, which means past the edge too: for the seconds the edge may still be behind, the owner's own clicks skip the CDN, or carry a parameter that makes the edge ask the origin. This is also why the API returns a 302: a browser may keep a 301 indefinitely, and then no edit would reach it. The edge still keeps a 302 for its short TTL, so clicks answered there never reach the origin: count them from the CDN's logs.

When things fail

The interviewer will ask what breaks. Answer for each box in turn:

They askTalk about
An app server diesreplicas behind the load balancer, which health-checks them and stops sending them traffic
The cache diesreads fall through to the database: up to 12 000 redirects a second at the peak (9 500 in the practice's run), more than the 8 000 one medium node takes, so give it a larger node or replicas for the minutes the cache needs to warm up
The database primary diesa replica promoted by failover, and what's lost if replication was asynchronous
Can a link be lost?synchronous commit, and synchronous replication if a lost machine mustn't lose links
A link goes viralskew lands on one key: the CDN and the cache absorb it, so the database sees almost none of it
Traffic doublesapp servers scale out; the cache and CDN carry the reads; writes are still light

Each answer names the failure, what the design does about it, and what it costs.

Features they add

Once the core holds, a strong candidate often gets a feature on top, to see whether the design bends or breaks. Each of these fits the design you have:

They askWhat to sayChapter
Custom aliases, like ink.ly/spring-saleThe alias is the code, written with an insert that fails if the code exists, so of two owners racing for one alias exactly one wins. Keep aliases out of the generated codes' way: make them a length generated codes never have, or let generated codes use the same insert and skip a code that's takenIdempotency
Links that expireStore the expiry with the link and check it on every read, so an expired link stops at once; a job deletes the rows later. The cache and the edge mustn't outlive it, so their TTL is capped at the time the link has leftChoosing a TTL
Click counts for each linkKeep the redirect path free of writes: log each click to a stream, and let a consumer add them up per link per hour in memory and write the totals to their own store, so a viral link isn't a hot row. Clicks the edge answered never reach the origin, so count those from the CDN's logsStreams, Hot keys

The pattern is the same each time: say where the new data lives, what the read path now checks, and what it costs the redirect.

Talking about tradeoffs

The same shape as always: I'd choose X, because of requirement Y; it costs Z; if W changed, I'd switch to V. With the whole design in front of you:

WeakStrong
"I'd add a cache.""Clicks are skewed, so a cache of the top links takes most reads off the database. The cost is stale targets after an edit, so the edit path invalidates the cache."
"I'd use NoSQL because it scales.""Every read is by code and nothing joins, so a key-value store fits. At 3 TB a relational database would also work; I'd pick whichever the team runs already."
"It's eventually consistent.""Other readers may see an edit a few seconds late, which is fine. The owner must see it straight away, so the edit purges the link from the CDN and the cache, and the owner's reads after an edit go to the primary."

In the wild

The ideas in this chapter, and where you'll meet them in real systems and interviews:

IdeaWhere you'll meet it
Real shortenersbit.ly, TinyURL, t.co
Redirect status codesRFC 9110 (HTTP Semantics), the section on redirection status codes, which defines 301 and 302
Short codes from rangesFlickr's "ticket servers" engineering post, on handing out IDs from a central counter
Purging an edge cacheCDN documentation on cache purging, such as Fastly's and Cloudflare's

Your practice

URL Shortener: build ink.ly with everything from the earlier chapters, then walk through it as you would in an interview: which requirement each box answers, and what each choice costs. The practice compresses the ratio: 5% of its requests are writes, about 19 redirects per write instead of 100, so the write path matters more than the estimate says. A relational database or a key-value store both work; say why you picked yours.

Key takeaways

  • Every box answers a requirement or a number: CDN for distance, cache for skew, ID generator for codes.
  • 62 characters over 7 places give about 3.5 trillion codes: short, and plenty.
  • An edit invalidates the cache, purges the edge, and sends the owner's next reads to the primary.
  • For each box, say what happens when it fails and what the design does about it.
  • For an added feature, say where its data lives, what the read path now checks, and what it costs the redirect.

Read first

  • Unique IDs
  • Durability
  • The Edge
  • Consistency
  • Data Modeling
  • The Design Interview

Practise it

  • URL Shortener: Mint short codes for long URLs and serve hot redirects under read-heavy load.
About · Contact · Privacy · Terms of use