When you send a prompt to a hosted model, a server runs the network once per output token, and it does so for thousands of other people at the same time. Whether that is cheap or ruinously expensive depends on a handful of engineering ideas. This guide explains LLM inference serving through the five papers that introduced those ideas, so you can read a provider's pricing page or a serving framework's benchmark claim with better questions.
Why one request at a time is wasteful
A language model produces text one token at a time. The Orca paper puts it directly: generative Transformers produce one output token per model iteration, so a single request needs many iterations.[2] A GPU does far more useful work when it processes many requests together, so servers batch them. The Kwon et al. paper states the consequence: high throughput requires batching enough requests at once.[1]
The older way to batch is to group requests, run them to completion and then start the next group. Orca's authors describe the problem with that: a request that finishes early cannot be returned to the client, and a newly arrived request has to wait for the whole current batch to finish.[2] Short answers are held hostage by long ones.
Continuous batching: change the batch every step
Orca's fix is iteration-level scheduling. The scheduler invokes the model for one iteration at a time rather than one whole request at a time, so the batch can change between iterations: finished requests leave and waiting ones join.[2] A complication is that not every operation in a Transformer can be batched across requests of different lengths, so the paper adds selective batching, which applies batching "only to a selected set of operations."[2]
The paper reports, for a GPT-3 175B model, about 36.9 times higher throughput than NVIDIA FasterTransformer at the same latency level.[2] Treat that as the authors' result against one specific baseline in 2022, not as a general speed-up you will see on your workload. The idea itself, however, is now the default thinking in serving systems.
The KV cache: what it saves and what it costs
Each new token has to attend to every earlier token. To avoid recomputing the attention keys and values of earlier tokens at every step, servers store them. This store is the KV cache. Kwon et al. describe each request's KV cache as large and changing dynamically in size.[1]
That is where the memory goes. Our own arithmetic, for an illustrative model and not any specific product: with 32 layers, 32 attention heads of dimension 128 and 16-bit values (2 bytes), one token needs 2 (keys and values) x 32 x 32 x 128 x 2 = 524,288 bytes, or 0.5 MiB. A single 4,096-token request would then hold about 2 GiB of cache, before counting the model weights. Multiply by the number of simultaneous users and you see why the cache, not the weights, often caps the batch size. The same logic is why long context windows are expensive to serve; see Long Context Windows: Capacity Is Not the Same as Reliability and Tokens, Context Windows and What an AI Request Really Costs.
PagedAttention: manage the cache like virtual memory
Kwon et al. argue that poor memory management wastes cache memory through fragmentation and redundant duplication, which limits batch size.[1] Their answer, PagedAttention, is an attention algorithm inspired by operating-system virtual memory and paging, and the vLLM system is built on it. The goals they state are near-zero waste of KV cache memory and flexible sharing of cache within and across requests.[1] They report 2 to 4 times higher throughput than systems such as FasterTransformer and Orca at the same latency, with larger gains for longer sequences, larger models and more complex decoding.[1] As with Orca, that is the authors' own comparison on their own setup.
Grouped-query attention: shrink the cache by design
PagedAttention manages the cache better; grouped-query attention (GQA) makes it smaller. In standard multi-head attention every query head has its own key and value head. Multi-query attention uses a single key-value head, which the GQA authors say "drastically speeds up decoder inference" but can degrade quality. GQA uses an intermediate number of key-value heads, more than one but fewer than the number of query heads, and the paper reports quality close to multi-head attention with speed comparable to multi-query attention.[5] They also describe a recipe to convert existing multi-head checkpoints using about 5% of the original pre-training compute.[5]
Applying that to our illustrative arithmetic: if the same model used 8 key-value heads instead of 32, the cache per token would be four times smaller, about 128 KiB instead of 512 KiB. That is a calculation, not a result from the paper, and real models differ.
Quantisation: fewer bits per weight
Weights are the other large memory consumer. Quantisation stores them in fewer bits. AWQ is a weight-only method whose authors argue that not all weights matter equally: it uses activation statistics, not weight values, to find salient channels and scales them before quantising, and reports that protecting about 1% of salient weights can greatly reduce quantisation error.[4] Their companion inference framework reports more than 3 times speed-up over the Hugging Face FP16 implementation on desktop and mobile GPUs.[4]
Read that carefully. The speed-up is against one named baseline, the scaling relies on a calibration set, and the results are the authors' own. Our interpretation is that quantisation is best treated as a trade-off to measure on your task, not a free lunch.
Speculative decoding: let a small model draft
Decoding is serial, so a large model spends most of its time waiting. Leviathan, Kalman and Matias propose having a smaller, cheaper model draft several tokens, then running the large model over the drafts in parallel and keeping what it agrees with. The method needs no retraining or architecture change, and uses a sampling scheme designed so the output distribution is unchanged. They report a 2 to 3 times acceleration on T5-XXL against the standard T5X implementation with identical outputs.[3]
Which technique fixes which problem
| Technique | Bottleneck it targets | What it costs or risks | Source |
|---|---|---|---|
| Iteration-level batching | GPU idle time, short requests waiting on long ones | Scheduler complexity; selective batching needed | Orca[2] |
| PagedAttention | Wasted KV cache memory, small batches | A more complex attention kernel and memory manager | Kwon et al.[1] |
| Grouped-query attention | Cache size per token | Must be in the model design or uptrained; possible quality change | Ainslie et al.[5] |
| Weight quantisation | Weight memory and bandwidth | Possible quality loss; needs calibration | AWQ[4] |
| Speculative decoding | Serial token generation | Needs a suitable draft model | Leviathan et al.[3] |
What this means when you buy or run a model
- Price differences are partly engineering. Providers that batch well and manage cache memory well can serve more users per GPU. We cannot see any provider's stack, so this is an inference, not a documented fact about any company.
- Long prompts cost more than they look. Every extra token adds to the cache and to attention work, which is why caching and routing matter; see Model Routing and Cascades.
- Reasoning models multiply the effect. More generated tokens mean more serial iterations; see Reasoning Models, Thinking Tokens and Cost.
- Read framework benchmarks as vendor claims. Each number above comes from the authors' own setup and a named baseline from 2022 or 2023; newer systems have moved on, and we did not test any of them.
Where to go next
If you are choosing between models, the practical lever is usually routing and prompt design rather than serving internals. If you are self-hosting an open-weight model, the techniques above are what frameworks like vLLM implement, and the questions to ask are about memory headroom, batch size and quality after quantisation. Evaluate on your own task before you trust any of them; How to Build LLM Evals for Your Application shows how.
