Appendix CGlossary

Terms the book uses, with the chapter that introduces each.

Bloom filter. A bitset per column chunk that can prove a value was never written to it. It has false positives and no false negatives. ch09

Coalescing. Merging nearby byte ranges into one request, reading the gap between them to save a request. ch10

Codec. The general-purpose compressor applied to every page of a column chunk after encoding, named in the chunk’s metadata: SNAPPY, GZIP, LZ4_RAW, ZSTD, BROTLI, or UNCOMPRESSED. ch07

Column chunk. One column’s data within one row group. Contiguous in the file. ch02

Column layout. Storing all of a column’s values together, then the next column’s. ch01

Column order. The footer’s statement, column by column, of which sort order min_value and max_value use. Without it their order is undefined. ch08

Compaction. Rewriting a table’s small files, and files with deletes, as a few whole files. It costs one large read and write, and makes later scans cheaper. ch15

Copy-on-write. Changing a row by rewriting the whole file that holds it. Writes are expensive; reads stay as they were. ch15

Data page version 2. A data page whose header records the level streams’ lengths, its rows and its nulls, and whose levels are never compressed. ch06

Definition level. For each value slot, how many of the optional and repeated fields on its path are present. A slot whose level is the column’s maximum holds a value. ch04

Dictionary encoding. Writing each distinct value once, in a dictionary page, and the data as small indices into it. ch05

Encoding. A type-aware way of writing a column’s values as bytes, such as PLAIN, RLE_DICTIONARY or DELTA_BINARY_PACKED. It happens before compression. ch05

Entropy coding. Giving frequent symbols shorter codes than rare ones, as DEFLATE’s Huffman codes do. ch07

Field id. A number a schema element may carry, which a table format assigns to a column and never changes or reuses, so that files written before a rename still match the column. ch16

Footer. The FileMetaData structure near the end of a file: schema, row counts, and the position and description of every column chunk. ch02

Footer length. The four-byte little-endian integer immediately before the closing magic. ch02

Hidden partitioning. Partitioning by a value a table format derives from a column, such as its day, and records in its metadata, rather than one a query must name and a path must hold. ch16

Hive-style partition. A directory named name=value holding a table’s files whose rows all share that value. The files omit the column; a reader takes it from the path. ch14

Key metadata. Bytes stored with an encrypted module that name the key protecting it without revealing it. ch13

Latency. The time from sending a request to the first byte of its response, paid once per request whatever its size. ch02, ch10

Little-endian. Storing a multi-byte integer with its least significant byte first. ch02

Logical type. What a physical type’s bytes mean: a date, a decimal, a string, a timestamp. ch03

LZ77. Compressing by replacing bytes the output already holds with how far back they are and how many. Every codec in ch07 is built on it.

Magic. The four ASCII bytes PAR1 at the start and end of every unencrypted Parquet file. ch02

Manifest. In Apache Iceberg, a file that lists a table’s data files or delete files, with each file’s partition values, row count and column statistics. ch16

Merge-on-read. Changing a row by writing a small file that describes the change, such as a position delete file, which every reader applies until compaction. ch15

Module. In modular encryption, one separately encrypted piece of a file: the footer, a page header or a page, each a length, a nonce, ciphertext and a tag. ch13

Object store. Storage addressed by key that serves whole objects and byte ranges over HTTP, such as S3. ch02

Page. The unit a column chunk is divided into for encoding and compression, with a header of its own. ch02

Page index. Two structures per column chunk, written after the row groups: the ColumnIndex, with each page’s bounds and null count, and the OffsetIndex, with each page’s position and first row. ch09

Physical type. How a value is stored: BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY or FIXED_LEN_BYTE_ARRAY. ch03

Position delete file. A Parquet file of file_path and pos: the rows, by their position in a data file, that a table no longer holds. ch15

Predicate pushdown. Evaluating a query’s condition against metadata before reading data, so that row groups and pages that cannot match are never read. ch09

Prefetch. Reading more of a file’s tail than the trailer, so that the footer arrives in the same request. ch02

Projection. Reading only the columns a query names. ch01

Query engine. Software that parses a query and runs it as a pipeline of stages, from reading data to returning an answer. ch12

Range request. An HTTP request for part of an object, Range: bytes=a-b, inclusive at both ends. ch02

Read amplification. Reading more than the answer needs, as a scan does when it fetches delete files and decodes rows they remove. ch15

Repetition level. For each value slot, the depth of the repeated field at which a new element starts; zero starts a new record. ch04

Row group. A horizontal slice of a table: some rows, all columns. ch02

Row layout. Storing each row’s values together. ch01

Snapshot. One version of a table: the list of data files and delete files that make it up. ch15

Sort order. The order a column’s statistics use, fixed by its type: signed, unsigned, by value, or byte by byte. ch08

Statistics. A column chunk’s or page’s minimum, maximum and null count, kept in its metadata so a reader can decide not to read it. ch08

Suffix range. A range request for the last n bytes of an object, Range: bytes=-n. The response reports the object’s size. ch02

Table format. A record, kept beside a table’s data files, of which files make up the table and what each holds: Delta Lake and Apache Iceberg are two. ch14

Thrift compact protocol. The binary serialisation Parquet uses for its footer and page headers. ch02

Trailer. The last eight bytes of a file: the footer length and the closing magic. ch02

Transaction log. Delta Lake’s table format: numbered JSON files of add and remove actions, each a version of the table, with each file’s partition values and statistics. ch14

Write amplification. Writing more than changed, as copy-on-write does when it rewrites a whole file for one row. ch15

Built from commit c32f48e.