Disaggregating Prefill and Decode

ch11 · Disaggregating Prefill and Decode¶

The problem¶

ch10 · Chunked Prefill and Scheduling Policy spent an entire chapter refereeing between prefill and decode: which gets the budget, in what order, at whose expense. That work exists only because both jobs run on the same hardware, in the same loop, competing for the same step.

They are not similar jobs. ch03 · The Arithmetic of Inference established that prefill is compute-bound and decode is memory-bandwidth-bound. We have been buying one machine to do well at two things that want different machines.

The idea¶

Run them in separate pools. Prefill a request in one, ship its keys and values to the other, decode there. The pools can then be sized independently — and, in a real deployment, provisioned on different hardware entirely.

The consequences are appealing:

• No interference. A long prefill cannot stall a streaming user, because it is not on the same device. The whole of ch10 · Chunked Prefill and Scheduling Policy becomes unnecessary.

• Independent scaling. Prefill-heavy traffic (ch21 · RAG and Long Context [DRAFT]’s RAG) and decode-heavy traffic can be scaled separately instead of by one batch-size dial.

• Hardware specialisation. Buy compute for the prefill pool and memory bandwidth for the decode pool, rather than compromising on both.

And one cost: every byte of KV cache has to cross between them.

    def _handoff(self, state: RequestState) -> None:
        """Move one sequence's KV cache from the prefill pool to the decode pool.

        Here that is a copy within one process. In a real deployment it is a network transfer, and
        its cost scales with the KV cache — which is why interconnect bandwidth, not compute,
        decides whether disaggregation is viable. A slow link turns every prefill into a stall at
        the other end.
        """

The build¶

The engine is a prefill pool, a handoff, and a decode pool:

        # -- prefill pool --------------------------------------------------------------
        room = self.max_batch_size - len(self.decoding)
        batch = self.waiting[: min(self.prefill_batch_size, max(room, 0))]
        if batch:
            self.waiting = self.waiting[len(batch) :]
            logits, caches = _prefill_batch(self.model, batch)
            for state, cache in zip(batch, caches, strict=True):
                state.past = cache
                self._handoff(state)
            outputs += self._emit(batch, _sample_batch(logits, batch))
            self.decoding += [s for s in batch if not s.finished]

        # -- decode pool ---------------------------------------------------------------
        ready = [s for s in self.decoding if s not in batch and not s.finished]
        if ready:
            outputs += self._emit(ready, _sample_batch(_decode_batch(self.model, ready), ready))

        return outputs

Two pool sizes rather than one — prefill_batch_size and max_batch_size — which is the entire architectural point, even though a single process cannot exploit it.

The measurement¶

Conditions: TinyGPT (reference, random weights) (5,838,080 params), 4x x86_64 CPU, torch 2.14.0+cu130, poisson, prompt 64-256, output 16-48, seed 7, arrival rate 16/s, measured 2026-09-13T15:29:49+00:00.

Roughly a wash, which is the expected result: the handoff is a memcpy within one process, and at this model size a request’s whole KV cache is well under a megabyte. Nothing here argues for or against the architecture.

Now the number that decides it. The handoff is not a design detail — it is the design. Here is what one request’s cache costs to move, across realistic interconnects:

Read the bottom row. An 8B model serving a 32k-token context has a KV cache of several gigabytes per request. Over ordinary datacentre Ethernet, moving it takes seconds — far longer than the prefill that produced it, and vastly longer than any latency budget. Over NVLink it takes about ten milliseconds, which is affordable.

That single comparison contains the whole chapter:

Disaggregation is an interconnect decision, not a scheduling one. It is an excellent architecture when prefill and decode sit on the same fabric, and unusable when they do not. No amount of scheduler cleverness compensates for a slow link.

It also explains why the technique appeared when it did. Disaggregated serving became practical once high-bandwidth interconnects between accelerators became ordinary; on commodity networking it would have been obviously absurd, and the papers would never have been written.

The cost¶

• Substantially more moving parts. Two pools, a transfer path, and failure modes that belong to neither — a prefill completing into a decode pool that has no room, a transfer failing midway, the pools disagreeing about who owns a sequence.

• A hard dependency on interconnect bandwidth, which is a procurement decision, not a software one.

• It is wrong below a certain scale. With one accelerator there is nothing to disaggregate, and the handoff is pure loss. Do not reach for this because it is modern.

• Prefix caching gets harder. ch09 · Prefix Caching’s cache lives in the prefill pool, but the blocks it saves are consumed in the decode pool. Sharing across a handoff is real work that our implementation does not attempt.

Key takeaways¶

• Prefill and decode are bound by different resources, so making one machine serve both is a compromise on both.

• Separating them removes interference by construction and allows the pools to be sized and specialised independently.

• The price is moving the entire KV cache between pools, once per request.

• That cost scales with model size and context length, and at production scale it is measured in gigabytes. Interconnect bandwidth, not scheduling, determines whether the architecture works.

• Below the scale where you have a fast fabric and enough traffic to justify two pools, disaggregation is pure overhead.

• A measurement that structurally cannot show a benefit is not evidence against it. Say so, and make the argument with arithmetic instead.

Looking ahead¶

Part III has taken the engine from one request at a time to a scheduler with paged memory, prefix reuse and an explicit prefill policy. Every remaining inefficiency is now in the mathematics itself rather than in how work is organised. ch12 · Attention at Speed [DRAFT] turns to attention: how to compute it with far less memory traffic, and how the architectures that shrink the KV cache change the arithmetic this entire part has been fighting.

Further reading¶

DistServe (Zhong et al., Appendix E) is the primary source and introduces the goodput framing this book has used since ch02 · Measuring What Matters; Splitwise covers the same ground with a stronger emphasis on hardware heterogeneity. Both are worth reading with this chapter’s handoff table beside them.