The API Surface

ch24 · The API Surface¶

The problem¶

Everything so far has been the engine. This chapter is the layer between it and an HTTP client, and that layer has an unusual property: it is where a surprising share of production incidents live, and none of them are visible in the engine’s own metrics.

A chat template with a stray value, a usage count that bills the wrong number, a stream that never terminates. Each is a small piece of ordinary data-handling code, each is easy to get wrong, and each is invisible to everything in ch25 · Observability for Serving Engines’s dashboard.

There is no web framework in this chapter. Request translation, the chat template, event framing and usage accounting are all ordinary data transformations, and they are where the interesting failures are. Bolting them to an ASGI server is the easy half and teaches nothing.

The measurement: a template change costs a fifth of the cache¶

The chat template decides the exact bytes the model sees:

class ChatTemplate:
    """How a list of messages becomes one string of tokens.

    This is the single most under-tested component in a serving stack. It decides the exact bytes
    the model sees, so a change to it changes the model's behaviour *and* — because prefix caching
    is content-addressed (chapter 9) — silently changes how much of the prompt can be reused. The
    ordering rule below is the one people get wrong.
    """

Because ch09 · Prefix Caching’s cache is content-addressed, the template also decides how much of the prompt can be reused. The same conversations, rendered three ways:

Conditions: TinyGPT (reference, random weights) (5,838,080 params), 4x x86_64 CPU, torch 2.14.0+cu130, 6 sessions x 4 turns, rendered three ways, no arrival process, measured 2026-09-13T19:11:52+00:00.

A timestamp in the system prompt costs a fifth of the reuse. It is one line, it looks helpful, it changes nothing about the model’s behaviour that anyone would notice, and it makes the system prompt different on every single request — so the cache misses on the very first block and everything after it.

The variant that moves the system prompt to the end is worse still and for the same reason: the shared text is no longer a prefix, and a prefix cache can only reuse a prefix. The content is identical; the position is what matters.

Both of these are the kind of change that ships in a pull request titled “add request context to system prompt”, passes review, passes tests, and shows up a week later as a cost regression nobody can attribute. Anything that varies per request belongs after the shared text, not before it — and ideally not in the prompt at all.

Streaming, and the way it is usually broken¶

A streaming response owns the incremental detokenizer, because that is the only place that can be correct: a multi-byte character split across two tokens must not become two replacement characters, and a stateless per-token decode cannot see that it is mid-character.

class StreamingResponse:
    """Turns an engine's token ids into the event stream a client reads.

    Owns the incremental detokenizer, because that is the only place that can be right: a
    multi-byte character split across two tokens must not be emitted as two replacement characters,
    and a stateless per-token decode cannot see that.
    """

The failure mode that matters is at the end of the stream:

def done_event() -> str:
    """The sentinel that ends the stream.

    Not optional, and not implied by the connection closing. A client that never sees it either
    hangs until its own timeout or reports a truncated response — and on the server side the
    request looks completed, so nothing alerts.
    """

A client that never sees the sentinel either hangs until its own timeout or reports a truncated response — and the server sees a completed request, so nothing alerts. That asymmetry is the whole reason this is worth a section: the failure is entirely on the client’s side of a boundary the server is not watching.

The same applies to cancellation. ch22 · Agents and Tool Use built abort into the engine and made the point there: a cancelled request still needs a terminal message. At this layer, that means a client disconnect must reach the engine — a disconnect that is noticed but not acted on is a request that keeps generating for nobody.

Usage accounting¶

class Usage:
    """What the caller is billed for.

    Counted from the token ids the engine actually processed, never from the text. Re-tokenizing
    the rendered prompt to count it is the classic way to bill for a different number than you
    served, because the count then depends on a code path the engine never ran.
    """

Counting from the token ids the engine processed, never from the text. Re-tokenizing the rendered prompt to count it is the classic way to bill a different number than you served, because the count then depends on a code path the engine never ran — and the two diverge exactly where tokenization is subtle, which is where the expensive prompts are.

This matters beyond billing. Usage counts are what every downstream quota, rate limit and capacity model is built on (ch19 · Multi-Tenancy and LoRA at Serving Time, ch27 · Cost and Capacity Planning), so a systematic error here propagates into decisions that look unrelated.

The rest of the surface, briefly¶

Four things this chapter builds no code for, in rough order of how often they cause an incident:

Backpressure. A server that accepts everything and queues it has moved the queue from the client to itself, where it is less visible and harder to shed. ch26 · Reliability and Operations is the answer.

Request size limits. An unbounded prompt is an unbounded prefill, which is an unbounded stall for everyone else (ch21 · RAG and Long Context). The limit belongs at the edge, in tokens rather than bytes, and it has to be checked before the request reaches the scheduler.

Timeouts that match the engine’s behaviour. A client timeout shorter than the queueing delay turns a slow period into a retry storm, and a retry storm turns a slow period into an outage.

Compatibility that is honest. An “OpenAI-compatible” endpoint that silently ignores parameters it does not implement produces output the caller did not ask for and cannot debug. Accepting a subset is fine; pretending is not:

class CompletionRequest:
    """The subset of ``/v1/chat/completions`` that changes what the engine does.

    Deliberately a subset. Half the fields in the real schema are either ignored by most servers or
    are aliases for each other, and pretending to support them is worse than not.
    """

The cost¶

• The template is now a performance-critical interface, owned by whoever writes prompts rather than whoever runs the server. It needs a test that pins the rendered bytes, or it will drift.

• Streaming makes every request stateful for its whole duration: a detokenizer, a connection and a place to send tokens. That is a per-request cost the engine does not see and a per-request leak if termination is ever missed.

• Cancellation plumbing crosses every layer — client disconnect, HTTP handler, engine abort — and is only correct if all three agree. ch22 · Agents and Tool Use lists the races.

• Usage counting is a correctness surface with money attached, which makes it the one place in this book where being approximately right is worse than being slow.

• Compatibility constrains the engine. Adopting somebody else’s API means adopting their parameters, including the ones that are awkward to implement well, and the pressure to accept and ignore them is constant.

Key takeaways¶

• The chat template is part of the serving stack. A per-request value in the system prompt costs a fifth of the prefix cache, and nothing in the engine’s metrics will say so.

• Anything that varies per request goes after the shared text. A prefix cache can only reuse a prefix, so position matters as much as content.

• A stream must end with an explicit sentinel. Silence is not termination, and the resulting failure is invisible from the server.

• Detokenize incrementally, in one place. Per-token decoding corrupts multi-byte characters.

• Count usage from the tokens the engine processed, not by re-tokenizing text.

• Limit request size in tokens at the edge. An unbounded prompt is an unbounded stall for everyone.

Looking ahead¶

Every failure in this chapter is invisible from the engine’s own metrics — which is a statement about the metrics. ch25 · Observability for Serving Engines builds the signals that would have caught these, and measures which one moves first when the engine gets into trouble.

Further reading¶

There is no good paper here; the material is API design and the sources are the specifications of the APIs people actually implement against. Read a real OpenAI-compatible server’s request handling — vLLM’s is readable — with attention to what it does with parameters it does not support, which is the honest part of the problem.