App server: what it does in a system design

Home › Reference › Components › App server

App server

Multi-replica compute with bounded concurrency and queue.

Instance tier: The machine size of each replica. Bigger tiers give more CPU cores and more requests in flight at once, but do not make a single request any faster.
Replicas: How many identical copies of the server run. More replicas raise total capacity and survive a single crash; each one has its own queue.
CPU cores / replica: CPU cores on each replica. Each core runs one processing burst at a time, so more cores means more requests computed in parallel. Blank uses the tier's default.
Max concurrency / replica: How many requests one replica handles at once, counting time spent waiting on other components. Extra requests wait in the queue. Blank uses the tier's default.
Max queue / replica: How many requests can wait for a free slot on each replica. When the queue is full, new requests are rejected at once. 0 means no waiting, only rejecting.
Processing time: How long the CPU work for one request takes, before any calls to other components. Pick the shape: constant, lognormal (mostly fast, some slow), exponential, or uniform.
Downstream pattern: The calls this makes to other components while handling a request, for example a database read. A request waits for each call to answer before it can finish.
Crashes: Scheduled crashes to test failure handling. While one is active, the target refuses requests, and anything it was holding is lost.