Inference gets put behind a load balancer, given an autoscaling policy on CPU, monitored with a p95 request latency, and treated as a stateless service with an unusually large container image. Every one of those choices is wrong in a specific, diagnosable way, and the reasons are worth knowing before the capacity review rather than during the incident.
One request is two workloads
Generating a response happens in two phases with opposite resource profiles, and almost every confusing measurement comes from averaging across them.
Prefill processes the prompt. All tokens are available at once, so the work parallelises well and saturates arithmetic units. It is compute-bound, and its cost scales with prompt length.
Decode generates the answer one token at a time. Each step depends on the previous one, so there is nothing to parallelise within a single sequence; the accelerator spends most of each step moving weights and cached state through memory. It is memory-bandwidth-bound, and its cost scales with the number of tokens produced.
| Prefill | Decode | |
|---|---|---|
| Bound by | Compute | Memory bandwidth |
| Parallel within one request | Yes, across prompt tokens | No, strictly sequential |
| Scales with | Prompt length | Output length |
| User-visible as | Time to first token | Tokens per second after that |
| Helped by batching | Somewhat | Substantially, it is what fills the bandwidth |
The immediate operational consequence: one latency number cannot describe this. A p95 that mixes a request producing 40 tokens with one producing 1,200 tokens is measuring output length, not service health. You need two SLOs, and they degrade for different reasons.
- Time to first token covers queue wait plus prefill. It rises when you are admitting more work than you can start.
- Inter-token latency covers decode. It rises when the batch is too full, and it is what makes a response feel like it is crawling even though it started instantly.
Those two move in opposite directions when you turn the batching dial, which is the whole tension of the system: more concurrency buys throughput and costs smoothness.
The cache is the constraint, and it is not a constant
To avoid recomputing attention over the whole sequence at every step, the server keeps per-token state for every sequence in flight. That cache is the binding resource, not the model weights: weights are loaded once and fixed, the cache grows and shrinks with traffic.
Its footprint is roughly proportional to the number of concurrent sequences multiplied by their length. Which means:
Maximum concurrency is not a number you configure. It is a function of the context lengths currently in flight.
This explains a failure that reads as inexplicable from the outside. Capacity was fine for months. A team shipped a feature that attaches more context to each request, average prompt length went from 800 tokens to 6,000, and the same request rate now exhausts memory. Nothing in the request-per-second graph moved. The unit of work changed underneath a metric that cannot see it.
So the capacity number worth tracking is not requests and not even tokens per second. It is cache occupancy, plus the distribution of context lengths that produced it. If you have one dashboard for inference, that is the one.
Admission control beats autoscaling
Scaling out is slow here in a way it is not for a normal service, because a new replica has to load a large set of weights before it can serve anything. Minutes, not seconds. Any policy that reacts to saturation by adding capacity will still be loading when the traffic spike ends.
That pushes the real control to the front door: decide what to accept, based on what it will cost you.
# Estimate the cache footprint before admitting, not after failing.
# Rejecting with a number the client can act on beats accepting and
# degrading everyone already in the batch.
def admit(req, state, limits):
# worst case: prompt now, plus everything we may generate
projected = req.prompt_tokens + req.max_output_tokens
need = projected * BYTES_PER_TOKEN
if state.cache_free < need:
wait = state.estimated_drain_time(need)
if wait > limits.max_queue_wait:
raise Overloaded(retry_after=wait, reason="cache")
return Queue(expected_wait=wait)
# protect smoothness for sequences already decoding
if state.active_sequences >= limits.max_batch_for_itl:
return Queue(expected_wait=state.estimated_slot_time())
return Admit()Two details matter more than the arithmetic. The estimate uses max_output_tokens, which means an unbounded output limit is a request for unbounded memory and should not be accepted; requiring clients to declare a cap is a capacity decision disguised as an API parameter. And the second guard exists because admitting one more sequence slows every sequence already generating. Beyond a point, accepting work makes the service worse for everyone rather than merely slower for the newcomer.
What to autoscale on, if you must
- Not CPU. The accelerator is doing the work. CPU tells you about tokenisation and serialisation.
- Not requests per second. A request is not a unit of work here, and the ratio changes without warning.
- Queue wait time is the honest demand signal, because it already accounts for how expensive the queued work is.
- Cache occupancy as the ceiling signal, with a headroom target well below full, since the cost of running out is dropped requests rather than slower ones.
- Keep a warm pool. Given the load time, the only fast scale-up is one that already happened. This is the same static-stability argument as removing “scale up” from step one of a runbook: capacity you need during an event has to exist before it.
Output length is an infrastructure lever
Because decode is sequential and bandwidth-bound while prefill parallelises, a generated token costs meaningfully more than a prompt token. That relationship is why provider pricing charges more for output than input, and it has a consequence people rarely connect to infrastructure.
Asking the model for a shorter answer is a capacity intervention. So is returning structured data rather than prose, refusing to restate the question, and dropping the closing paragraph that summarises what was just said. A prompt change that cuts average output by a third does more for cost and for tail latency than most tuning work, and it usually improves the product at the same time, because nobody wanted the summary of the summary.
That is the useful shape of this whole subject: the levers are not exotic. Bound the inputs, bound the outputs, measure the two latencies separately, watch the cache rather than the request count, and keep capacity warm because you cannot conjure it. None of that requires understanding the model. It requires refusing to treat it as a web server.