Appendix BThe fixtures

Fixtures#

Generated by fixtures/generate.py. Do not edit this file or the fixtures by hand: change the generator and run make fixtures. make check fails if a committed file differs from what the generator writes.

Every fixture is written by pyarrow, a production Parquet implementation, and never by this repository’s own code. The .json beside each file is pyarrow’s own description of it, and the Rust tests use it as an oracle.

tiny.parquet#

The smallest useful file: four rows of a sales table, one row group, no compression, no dictionary, every column required so no levels are stored. Small enough to read completely by eye. The footer laboratory starts here.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

multiple-row-groups.parquet#

The same sales schema split into row groups of two rows. The footer now describes several row groups, each with its own column chunks and statistics, so the structure view has more than one branch.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False
row_group_size=2

types.parquet#

Three orders with a column for each physical type and the common logical types: a signed and an unsigned small integer, a decimal, a date, a UTC timestamp, a nullable string and an optional group. The same bytes mean different values depending on the logical type the schema gives them, which is ch03’s subject.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

nested.parquet#

Four orders whose fields are missing in every way a nested value can be: a null string, a null list, an empty list, a list holding a null, and a list of structs each holding a list of its own. Every column stores repetition and definition levels, which is ch04’s subject.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

dictionary.parquet#

Twelve orders written with dictionary encoding, the writer’s default. Each column chunk starts with a dictionary page of its distinct values, and its data page holds small indices into it, packed with the RLE / bit-packing hybrid.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

encodings.parquet#

Forty orders with one column per encoding a writer uses instead of a dictionary: DELTA_BINARY_PACKED for increasing integers, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY for strings, and BYTE_STREAM_SPLIT for floats. Each column’s values were chosen to suit its encoding.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False
column_encoding={'order_id': 'DELTA_BINARY_PACKED', 'ordered_at': 'DELTA_BINARY_PACKED', 'sku': 'DELTA_LENGTH_BYTE_ARRAY', 'url': 'DELTA_BYTE_ARRAY', 'weight_kg': 'BYTE_STREAM_SPLIT'}

pages.parquet#

Sixty orders written in small pages, so every column chunk holds several data pages, and the country column has a dictionary page in front of them and nulls in them. Data page version 1.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=['country']
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False
data_page_size=128
write_batch_size=16

pages-v2.parquet#

The same sixty orders in data page version 2, with a CRC-32 checksum in every page header. Compare its pages with pages.parquet’s: the levels move out of the compressible section and each header counts its rows and nulls.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=['country']
write_statistics=True
store_schema=False
data_page_version='2.0'
write_page_index=False
data_page_size=128
write_batch_size=16
write_page_checksum=True

codec-none.parquet#

256 orders, PLAIN-encoded with no dictionary and not compressed: the baseline the other codec-* files are measured against.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

codec-snappy.parquet#

256 orders, PLAIN-encoded with no dictionary, compressed with snappy. The codec-* files hold identical pages before compression, so any difference in their sizes is the codec’s.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

codec-gzip.parquet#

256 orders, PLAIN-encoded with no dictionary, compressed with gzip. The codec-* files hold identical pages before compression, so any difference in their sizes is the codec’s.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='gzip'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

codec-lz4.parquet#

256 orders, PLAIN-encoded with no dictionary, compressed with lz4. The codec-* files hold identical pages before compression, so any difference in their sizes is the codec’s.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='lz4'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

codec-zstd.parquet#

256 orders, PLAIN-encoded with no dictionary, compressed with zstd. The codec-* files hold identical pages before compression, so any difference in their sizes is the codec’s.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='zstd'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

codec-brotli.parquet#

256 orders, PLAIN-encoded with no dictionary, compressed with brotli. The codec-* files hold identical pages before compression, so any difference in their sizes is the codec’s.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='brotli'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

statistics.parquet#

Twelve orders in three row groups, with a column for each way a reader can misread statistics: unsigned integers above 2^31, negative small integers, strings with non-ASCII bytes, floats with NaN and signed zeros, negative decimals, a column that is null throughout one row group, and a column written without statistics. The row groups declare that they are sorted by order_id.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=['order_id', 'customer_id', 'delta', 'city', 'temp_c', 'amount', 'coupon']
store_schema=False
data_page_version='1.0'
write_page_index=False
row_group_size=4
sorting_columns=[{'column_index': 0, 'descending': False, 'nulls_first': False}]

pruning-sorted.parquet#

800 orders written in order_id order: four row groups of 200, pages of 40 rows, a page index for every column, and a Bloom filter for customer_id. Because the rows are sorted, each row group and page covers a narrow range of order_id, and a reader can skip most of the file for a query on it.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=['country']
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40
bloom_filter_options={'customer_id': {'ndv': 200, 'fpp': 0.05}}

pruning-shuffled.parquet#

The same 800 orders as pruning-sorted.parquet, shuffled, and written the same way. Every row group and page now covers nearly the whole range of order_id, and the statistics can rule out almost nothing.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=['country']
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40
bloom_filter_options={'customer_id': {'ndv': 200, 'fpp': 0.05}}

writing-baseline.parquet#

800 orders in order_id order, written the way ch11 starts from: Snappy, dictionary encoding, four row groups of 200, pages of 40 rows, and a page index. Every other writing-* file changes one of these.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40

writing-one-group.parquet#

writing-baseline.parquet in one row group of 800 rows.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=800
max_rows_per_page=40

writing-small-groups.parquet#

writing-baseline.parquet in row groups of 40 rows: twenty row groups, each with its own column chunks, statistics and page index.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=40
max_rows_per_page=40

writing-by-country.parquet#

writing-baseline.parquet’s rows sorted by country, then order_id.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40

writing-shuffled.parquet#

writing-baseline.parquet’s rows in a pseudo-random order.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40

writing-plain.parquet#

writing-baseline.parquet without dictionary encoding.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=True
row_group_size=200
max_rows_per_page=40

writing-no-index.parquet#

writing-baseline.parquet without a page index. pyarrow then writes each page’s statistics into its page header instead.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=True
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False
row_group_size=200
max_rows_per_page=40

encrypted-footer.parquet#

Twelve orders with modular encryption and an encrypted footer: the file ends in PARE. email is encrypted with the pii key and amount_cents with the finance key; order_id and country are not encrypted, but the footer that would say where they are is. Written once, since encryption is never byte-for-byte reproducible; the check decrypts it with the published test keys.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

plaintext-footer.parquet#

The same orders and keys with a plaintext footer: the file ends in PAR1, the footer is readable and signed, and only the two encrypted columns’ pages and details are hidden.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='none'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False

pages-v2-snappy.parquet#

pages-v2.parquet compressed with Snappy. In data page version 2 only a page’s values are compressed: its levels stay as they were, so a reader can count rows and nulls without decompressing anything.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='snappy'
use_dictionary=['country']
write_statistics=True
store_schema=False
data_page_version='2.0'
write_page_index=False
data_page_size=128
write_batch_size=16
write_page_checksum=True

codec-zstd-split.parquet#

codec-zstd.parquet with its two float columns written BYTE_STREAM_SPLIT. Comparing the two files measures what splitting a float’s bytes is worth to a compressor, for decimal-looking floats and for noisy ones.

Leaf columns, as pyarrow reads the Parquet schema:

Writer options:

compression='zstd'
use_dictionary=False
write_statistics=True
store_schema=False
data_page_version='1.0'
write_page_index=False
column_encoding={'weight_kg': 'BYTE_STREAM_SPLIT', 'distance_km': 'BYTE_STREAM_SPLIT'}