Queries, operator by operator

Build a query engine to learn what a query costs.

A hands-on book for data engineers who run queries every day and want to know what they cost. You build a small query engine in Python on Apache Arrow, one operator at a time, and hold every operator to DuckDB: the rows it moves, the bytes it reads, the memory it holds. Before each measurement you predict it. Every number in the book is computed by an engine when the book is built, and every panel can run again in your browser. Start with the Preface to see how a chapter works and what you need.

A plan of three operators, scan, filter and project, with rows flowing up between them, and for each operator a prediction, an estimate and a measurement drawn as bars.

By Chris Snow, in collaboration with Claude (Anthropic)

The prose and figures are under CC BY-NC 4.0; the code is under Apache 2.0.

Built from commit dc21108.