Appendix BThe fixtures
Every dataset in the book is generated by fixtures/generate.py from a fixed seed and written
by pyarrow, a production Parquet writer. Each exists to show one property: an order, a skew, a
number of distinct values. The same commit writes the same bytes everywhere, and the build fails
if a committed file differs from what the generator writes.
| File | Rows | Row groups | Bytes | Why it exists |
|---|---|---|---|---|
countries.parquet | 12 | 1 | 1,077 | The countries customers.country names, each with its region: the smallest table of ch13’s three-way join, whose region a query filters on. |
customers.parquet | 1,000 | 1 | 18,117 | The dimension table orders.customer_id points into: one row per customer, small enough to be the build side of any join. |
orders-by-month.parquet | 20,000 | 12 | 513,221 | The sorted orders as a table of twelve monthly files in Hive’s layout, with the table’s metadata listing each file and the bounds of every column. A reader of the metadata can skip a file without opening it; a reader of the files alone must open each footer. |
orders-paged.parquet | 20,000 | 1 | 415,368 | The sorted orders in one row group, with a page index and small pages, and a dictionary only for the columns with few distinct values. Its one row group spans the year, so row group statistics can skip nothing; its pages each hold a few weeks, so a reader that uses the page index can skip most of them. |
orders-shuffled.parquet | 20,000 | 10 | 534,245 | The same orders in a seeded random order. Every row group spans the whole year, so no filter on the date can skip one: the control for orders-sorted. |
orders-sorted.parquet | 20,000 | 10 | 497,857 | Orders written in date order. Each row group holds a narrow run of dates, so a filter on the date can skip most row groups, and neighbouring rows are neighbours in memory. |
Read from the manifests fixtures/generate.py wrote, at build time.
Each file has a manifest beside it (fixtures/<name>.json) holding what pyarrow and DuckDB
report about it: its row groups and their statistics, the distinct and null counts of each
column, and the properties the generator promises. Tests check the files against the manifests
and the promises.
Built from commit dc21108.