Parquet, byte by byte · Build a Parquet reader while you read about one

Parquet, byte by byte

What is in a Parquet file, read from its bytes, with a working reader to show for it.

A book for data engineers who write Parquet every day and want to know what is in the files. You learn the format by reading real files byte by byte, in Python and in Rust, and you build a working Parquet reader as you go. Then you ask pyarrow and the Rust parquet crate the same questions and check their answers against yours. Everything runs in the page: the code you read by hand, the problems and their tests, and the pictures the reader draws. Start with the Preface to see how a chapter works and what you need to run the code at a desk.

A Parquet file drawn as a strip of bytes from its first byte to its last: the PAR1 magic, two row groups each holding a column chunk per column, the footer, the footer's length, and the PAR1 magic again. An arrow from the length leads back to the start of the footer, and arrows from the footer lead to the start of every column chunk.

By Chris Snow, in collaboration with Claude (Anthropic)

The prose and figures are under CC BY-NC 4.0; the code is under Apache 2.0.

Built from commit c32f48e.