Build a production inference engine, one measurement at a time.
Why this book¶
Most material on LLM inference is either a paper describing one technique in isolation, or documentation telling you which flags to pass to vLLM. Neither teaches you how a serving engine works, and neither lets you feel why each optimisation exists.
So we build one. We start with a hundred-line server that wraps model.generate() and serves
roughly one request per second, and we take it apart and rebuild it: KV cache, continuous
batching, a paged block manager, prefix caching, chunked prefill, speculative decoding,
quantisation, tensor parallelism, an OpenAI-compatible API, and the observability you need to
run the result.
By the end you will have an engine of your own, you will be able to read the vLLM and SGLang source with confidence, and you will be able to size and price a deployment.
How to read it¶
Two things hold the book together.
One mental model. Prefill is compute-bound; decode is memory-bandwidth-bound. Chapter 3 derives this, and every later chapter says which of the two bounds its technique attacks and what that costs. If you remember one thing from this book, that is the thing.
One scorecard. Chapter 2 builds a benchmark harness, and every chapter that changes the engine ends by re-running it and appending a row. Progress is measured, never asserted. Every number in this book is generated by a committed script and stamped with the hardware, model and library versions that produced it — so you can reproduce it, and so you can tell when it has gone stale.
Read Parts I–III in order; they build on each other directly. After that you can follow your own interests — each chapter states its prerequisites.
You will need¶
Almost nothing. The main arc of the book runs on any laptop, CPU-only, using a small model. Absolute numbers will not match a datacentre GPU, but the mechanisms and the ratios do, and that is what the book is teaching.
Two chapters need a GPU and say so in their headers. Nothing depends on them. See Appendix B to check what your machine can run before you start.
What you should already know¶
Comfortable Python, some PyTorch, and a working idea of what a transformer is — self-attention, layers, logits. You do not need CUDA, distributed systems, or queueing theory; each is introduced where it is needed.
If the model itself is the fuzzy part, the LLM From Scratch series builds a GPT from first principles and is the natural prerequisite to this book.