Some work has to happen, but nobody has to wait for it. Moving that work out of the request, into a queue that workers drain in the background, keeps pages fast however slow the work is.
Now or later
A writer who asks for a PDF export needs to hear "we're on it" right away; the file can arrive a few seconds later. The same goes for sending emails, resizing pictures and updating search results. Doing that work inside the request makes the user wait for all of it, and holds a thread the whole time.
Queues and workers
A message queue stores each job and acknowledges it in a couple of milliseconds, so the request is done. Workers take jobs from the queue, as many as they have room for, and do the work. The jobs nobody has taken yet wait in the queue, roughly oldest first: many queues (Amazon SQS standard queues, for one) only keep order on a best-effort basis.
The request ends at the queue. The work happens behind it.
Because the queue keeps the job durably, it's a safe place to acknowledge: a stored job will be done even if a worker crashes halfway.
Saving and queuing together
Often a request does two things: it saves a row in the database, such as the export request, and queues the job that follows from it. Those are two systems, and the app can crash between them. Save first and crash before queuing, and the job is lost: the database says an export was asked for, and no worker will ever render it. Queue first and crash before saving, and a worker runs a job for a row that doesn't exist.
The usual fix is a transactional outbox. The app writes the job as a row in an outbox table, in the same database transaction as the data, so both are saved or neither is. A separate relay reads new outbox rows, publishes them to the queue, and marks them sent. A relay that crashes after publishing and before marking publishes the row again, so this is at least once too, and the workers still have to be idempotent.
The job is saved with the data, and reaches the queue a moment later.
Change data capture is the other way to do it: a tool such as Debezium reads the database's own log of committed changes and publishes them. Nothing extra goes into the transaction, and there's one more system to run.
Sizing workers
Workers need enough capacity for the average rate of jobs, not the peak, because nobody is waiting on the result: during a burst, jobs simply wait in the queue a little longer. That holds as long as the backlog from the busiest stretch drains within the time a job is allowed to take; if exports must arrive within a minute, the workers must clear any burst in under a minute. The arithmetic is the one from scaling out: jobs per second times CPU seconds per job gives the cores you need. Say 30 thumbnails a second at 0.5 s of CPU each: that keeps 15 cores busy, four medium instances at about 94%, so five for headroom (75% busy, which a queue can absorb).
Each job needs a core while it runs, so "Jobs per instance" should match the tier's cores (the default). For jobs that keep the CPU busy, like rendering a PDF, raising it doesn't add capacity: the extra jobs just share the same cores, so every job takes longer. Jobs that mostly wait on a network or a disk (sending emails, calling another service) are different: they leave the core free while they wait, so more jobs per core does help.
Retries and dead letters
Jobs fail: a render crashes, a service is briefly down. The queue keeps the job and offers it again after a backoff, up to a set number of attempts. A job that fails every attempt goes to a dead-letter queue for someone to look at.
At least once
The queue can't see whether a worker is still busy or has died. It gives each worker a visibility timeout: if the job isn't done in time, the queue hands it out again. Set it shorter than the job takes and slow jobs run twice.
Even with a sensible timeout, a crash at the wrong moment can make a job run twice. Queues deliver at least once, so jobs should be idempotent, safe to run twice (the Idempotency chapter has the details), like saving the same PDF under the same name. That makes a repeat harmless, not free: the work still runs twice, so the timeout still has to fit the job.
Follow one job through, with a worker that needs 1.5 seconds. Change the timeout, make the save ignore repeats, then crash the first worker and see what the queue is for.
One event, many consumers
A job queue hands each job to one worker. Often one event needs several reactions: a new post should be indexed for search, sent to followers and counted for analytics. Publish/subscribe (pub/sub) sends each published message to every subscriber, so each service gets its own copy and works at its own pace. A log (stream) goes further: it keeps the messages, in order, so a new service can read history it never saw.
| Job queue | Pub/sub topic | Log (stream) | |
|---|---|---|---|
| Each message goes to | one worker | every subscriber | every consumer group (a set of workers that share one reading of the log) |
| After it's handled | deleted | deleted per subscriber | kept, and can be replayed |
| Typical use | exports, emails | fan-out of events | event history |
The Pub/sub and Streams chapters, later on, build and measure both.
In the wild
The ideas in this chapter, and where you'll meet them in real systems and interviews:
| Idea | Where you'll meet it |
|---|---|
| Job queues | Amazon SQS (visibility timeouts, dead-letter queues), RabbitMQ |
| Workers | Celery, Sidekiq, AWS Lambda reading from a queue |
| Logs and pub/sub | Kafka, Amazon Kinesis, Amazon SNS, Google Pub/Sub |
| Outbox and change data capture | An outbox table read by a relay, or Debezium streaming PostgreSQL's or MySQL's log into Kafka |
The simulator's queues cost a few cents an hour, however busy. Real ones bill per request: Amazon SQS charges about $0.40 per million, and each message takes a send, a receive and a delete, so 40 000 messages a second costs well over $100 an hour unless you batch them, up to ten to a request.
Your practice
Inkwell's exports are back: every export is now rendered as a finished book, about 400 ms of CPU each, 50 a second. Writers should hear "we're on it" at once and get their PDF within seconds, and a few renders fail and need another try. Queue the exports, and size the workers that render them, for as little as you can.