ch02Anatomy of a Parquet file

What you build: Footer discovery over a simulated object store: size, trailer, footer range, FileMetaData.

The question#

Given only a file’s bytes, how does a reader find anything in it?

ch01 showed why a table is worth storing one column at a time. It also left a problem. A column layout scatters each row across the file, so a reader cannot scan from the start and pick values up as it goes. It needs a map: which bytes hold which column, for which rows. A Parquet file carries that map, and it carries it at the end.

First you will read a file’s bytes yourself, a few at a time. Then you will follow the book’s reader, which knows one thing: the name of an object in object storage. You will watch every request it makes, see the bytes each request returns, and then build the code that made those requests.

The experiment#

Read the bytes yourself#

tiny.parquet is a real Parquet file written by pyarrow: four orders from a sales table, and Appendix B says exactly how it was written. Before any reader opens it, open it yourself. Each step below is a few lines of code. In Python, run them in the page, change them with Edit, and run them again. In Rust, open them in a Codespace, or run one at a desk with cargo run -p walkthroughs --bin and its name.

The first bytes. A file format usually announces itself at the start.

# Read the whole file, and look at its first four bytes.
data = open("fixtures/tiny.parquet", "rb").read()

print(len(data), "bytes in the file")
print("the first four:", data[:4])

//! Read the whole file, and look at its first four bytes.

fn main() {
    let data = std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");

    println!("{} bytes in the file", data.len());
    println!("the first four: {:?}", String::from_utf8_lossy(&data[..4]));
}

Four ASCII letters, PAR1: Parquet’s magic number, which says what kind of file this is.

The last bytes. Now the other end of the file.

# The last four bytes, and whether they match the first four.
data = open("fixtures/tiny.parquet", "rb").read()

print("the last four:", data[-4:])
print("the same as the first four:", data[-4:] == data[:4])

//! The last four bytes, and whether they match the first four.

fn main() {
    let data = std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");
    let last = &data[data.len() - 4..];

    println!("the last four: {:?}", String::from_utf8_lossy(last));
    println!("the same as the first four: {}", last == &data[..4]);
}

The same four letters. A Parquet file carries its magic at both ends, and the end is the one a reader starts from, as you will see.

The four bytes before them. These are not letters. Read them as a number, least significant byte first, which is what little-endian means:

# The four bytes before the closing magic, read as a little-endian number.
data = open("fixtures/tiny.parquet", "rb").read()

length_bytes = data[-8:-4]
print("the four before them:", length_bytes.hex(" "))
print("read little-endian:", int.from_bytes(length_bytes, "little"))

//! The four bytes before the closing magic, read as a little-endian number.

fn main() {
    let data = std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");

    let length_bytes: [u8; 4] = data[data.len() - 8..data.len() - 4].try_into().unwrap();
    println!("the four before them: {}", hex(&length_bytes));
    println!("read little-endian: {}", u32::from_le_bytes(length_bytes));
}

/// Bytes as two hex digits each, spaced, as Python's `bytes.hex(" ")` writes them.
fn hex(bytes: &[u8]) -> String {
    bytes
        .iter()
        .map(|b| format!("{b:02x}"))
        .collect::<Vec<_>>()
        .join(" ")
}

That number is the length of the footer: the file’s map of everything else in it.

Where the footer is. The footer ends where the last eight bytes begin, and it is as long as they say, so it starts that many bytes earlier:

# Where the footer starts: it ends where the last eight bytes begin, and is as long as they say.
data = open("fixtures/tiny.parquet", "rb").read()
footer_length = int.from_bytes(data[-8:-4], "little")

footer_start = len(data) - 8 - footer_length
footer = data[footer_start : len(data) - 8]
print("the footer starts at byte", footer_start, "and is", len(footer), "bytes long")
print("its first eight bytes:", footer[:8].hex(" "))

//! Where the footer starts: it ends where the last eight bytes begin, and is as long as they say.

fn main() {
    let data = std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");
    let length_bytes: [u8; 4] = data[data.len() - 8..data.len() - 4].try_into().unwrap();
    let footer_length = u32::from_le_bytes(length_bytes) as usize;

    let footer_start = data.len() - 8 - footer_length;
    let footer = &data[footer_start..data.len() - 8];
    println!(
        "the footer starts at byte {footer_start} and is {} bytes long",
        footer.len()
    );
    println!("its first eight bytes: {}", hex(&footer[..8]));
}

/// Bytes as two hex digits each, spaced, as Python's `bytes.hex(" ")` writes them.
fn hex(bytes: &[u8]) -> String {
    bytes
        .iter()
        .map(|b| format!("{b:02x}"))
        .collect::<Vec<_>>()
        .join(" ")
}

Those bytes are the footer. The rest of this chapter follows a reader to them and decodes their Thrift fields. ch03 reads the schema out of them.

Break it. Change one byte of the length and do the same arithmetic:

# Damage the footer length, and see where the arithmetic now puts the footer.
data = bytearray(open("fixtures/tiny.parquet", "rb").read())
data[-7] = 0xFF  # the second of the four length bytes

footer_length = int.from_bytes(data[-8:-4], "little")
print("the footer length now reads", footer_length)
print("so the footer would start at byte", len(data) - 8 - footer_length)

//! Damage the footer length, and see where the arithmetic now puts the footer.

fn main() {
    let mut data =
        std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");
    let n = data.len();
    data[n - 7] = 0xFF; // the second of the four length bytes

    let length_bytes: [u8; 4] = data[n - 8..n - 4].try_into().unwrap();
    let footer_length = u32::from_le_bytes(length_bytes);
    println!("the footer length now reads {footer_length}");
    // Signed arithmetic: the answer can be negative, and that is the point.
    println!(
        "so the footer would start at byte {}",
        n as i64 - 8 - i64::from(footer_length)
    );
}

The arithmetic cannot tell that anything is wrong: it puts the footer before the start of the file. A reader has to check what it reads before it trusts it, and the book’s reader below refuses this very damage.

Try changes of your own with Edit: read the first eight bytes, print the whole footer, or open another fixture, such as fixtures/multiple-row-groups.parquet.

Open a file with the book’s reader#

Now let the book’s reader do what you have done. It sees the file only through a store that logs every request, as an object store would bill them, so it cannot cheat by reading the whole file:

# The book's reader opens the file as you did, and the store it reads from logs every request.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab.object_store import MemoryStore, NetworkModel, TracingStore
from parquet_lab.reader import FooterOptions, Head, SuffixRange, read_footer

files = MemoryStore()
files.put("tiny.parquet", open("fixtures/tiny.parquet", "rb").read())
store = TracingStore(files, NetworkModel())

# Try SuffixRange() for Head(), then raise prefetch until the footer comes in the first read.
opened = read_footer(store, "tiny.parquet", FooterOptions(size=Head(), prefetch=8))

for r in store.requests:
    print(r.method, r.range or "·", f"-> {r.bytes_returned} bytes by {r.end_us // 1000} ms:", r.why)
print("footer length:", opened.trailer.footer_length, "rows:", opened.metadata.num_rows)

//! The book's reader opens the file as you did, and the store it reads from logs every request.

use parquet_lab::object_store::{MemoryStore, NetworkModel, TracingStore};
use parquet_lab::reader::{read_footer, FooterOptions, SizeSource};

fn main() {
    let mut files = MemoryStore::new();
    let data = std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");
    files.put("tiny.parquet", data);
    let mut store = TracingStore::new(files, NetworkModel::default());

    // Try SizeSource::SuffixRange for Head, then raise prefetch until the footer comes in the first read.
    let options = FooterOptions {
        size: SizeSource::Head,
        prefetch: 8,
    };
    let opened = read_footer(&mut store, "tiny.parquet", options).expect("a Parquet file");

    for r in &store.requests {
        let range = r.range.as_deref().unwrap_or("·");
        let (method, bytes, ms) = (r.method.name(), r.bytes_returned, r.end_us / 1000);
        println!("{method} {range} -> {bytes} bytes by {ms} ms: {}", r.why);
    }
    println!(
        "footer length: {} rows: {}",
        opened.trailer.footer_length, opened.metadata.num_rows
    );
}

Three requests: the size, then the trailer, then the footer, each waiting for the one before. Every byte the reader did not ask for is a byte it never saw, and that is most of the file: it has learned the schema, the row count and where every column lives without reading a single value. Change the call and run it again. Ask with a suffix range instead of HEAD, and the first request disappears, because the response to a suffix range says how large the file is. Raise prefetch until the first read takes in the footer too, and the third request disappears; the trace shows what the bigger read cost in bytes.

The reader also checks what it reads before it trusts it. Hand it the file you damaged by hand above:

# The same damage as before, handed to the book's reader: it checks the length before trusting it.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab.object_store import MemoryStore, NetworkModel, TracingStore
from parquet_lab.reader import FooterOptions, read_footer

data = bytearray(open("fixtures/tiny.parquet", "rb").read())
data[-7] = 0xFF  # the second byte of the footer length
files = MemoryStore()
files.put("tiny.parquet", bytes(data))
store = TracingStore(files, NetworkModel())

try:
    read_footer(store, "tiny.parquet", FooterOptions())
except Exception as refused:
    print("the reader refuses:", refused)
print("after", len(store.requests), "requests:", ", ".join(r.why for r in store.requests))

//! The same damage as before, handed to the book's reader: it checks the length before trusting it.

use parquet_lab::object_store::{MemoryStore, NetworkModel, TracingStore};
use parquet_lab::reader::{read_footer, FooterOptions};

fn main() {
    let mut data =
        std::fs::read("fixtures/tiny.parquet").expect("run this from the repository's root");
    let n = data.len();
    data[n - 7] = 0xFF; // the second byte of the footer length
    let mut files = MemoryStore::new();
    files.put("tiny.parquet", data);
    let mut store = TracingStore::new(files, NetworkModel::default());

    if let Err(refused) = read_footer(&mut store, "tiny.parquet", FooterOptions::default()) {
        println!("the reader refuses: {refused}");
    }
    let why: Vec<&str> = store.requests.iter().map(|r| r.why.as_str()).collect();
    println!(
        "after {} requests: {}",
        store.requests.len(),
        why.join(", ")
    );
}

It stops after the trailer, before asking the store for a byte of a footer that cannot exist, and says why. Damage the magic instead, data[-1], and it refuses for a different reason.

What the trailer says#

A Parquet file ends with eight bytes: a four-byte footer length, then the four ASCII bytes PAR1. These are the last eight bytes of tiny.parquet:

The footer ends where the trailer begins, and it is as long as the trailer says. That is all a reader needs to find it.

Why the reader made the requests it did#

With the default settings the reader follows the procedure literally. It does not know how large the file is, so it asks. It cannot know how long the footer is until it has read the trailer, so it reads the trailer. Only then can it ask for the footer:

Every request waits for the one before it, because each needs a number the previous one returned. On object storage the wait is the expensive part. A request takes a long time to start and then moves bytes quickly, so three small requests cost far more than one large one.

Real readers shorten the chain in two ways, the two changes you made to the call above:

• The size is often known already. A directory listing or a table format’s manifest records each file’s size (ch14). A suffix range, Range: bytes=-n, asks for the last n bytes without knowing the size, and the response reports it.
• Read more than the trailer. If the first read is large enough to hold the footer too, the third request never happens. The cost is bytes the reader may not use.

Here is what each combination costs on both fixtures, computed by the same reader:

The last row for each file is the pattern most production readers use: one suffix read of a few tens of kilobytes, on the bet that the footer fits. For these files it does. A footer grows with every row group, and ch11 writes one table with small row groups and shows its footer many times larger than with one row group.

The whole file#

The footer is a map. This panel draws what it describes: every region of the file, as the reader parsed it. Click a region in the structure view to see its bytes, or click a byte to see which structure it belongs to. The dim bytes are the ones the reader never asked for when it opened the file above.

The same regions, measured by the reader:

Read the structure view from the top, and the file’s design appears:

• The opening magic is the same four bytes as the closing one. It lets a tool that reads from the front recognise the file. A reader working from the end does not need it, and the reader in this book does not request it.
• A row group is a horizontal slice of the table: some number of rows, all columns. A writer holds one row group in memory at a time, arranges it by column, and writes it out. Row groups are also the unit a reader can skip or hand to another worker. tiny.parquet has one row group. Switch the panel to multiple-row-groups.parquet to see several.
• A column chunk is one column’s data within one row group. It is contiguous, so reading one column of one row group is one range request.
• A page is the unit a column chunk is divided into for encoding and compression. Each page starts with a header that says how long its body is. In tiny.parquet each column chunk is a single page, and the body holds the four values in plain little-endian form. Select the first value byte of order_id’s page body and read it as a 64-bit integer.
• The footer is a FileMetaData structure: the schema, the row count, and for every column chunk its offsets, sizes, encodings, compression and statistics.

The footer is at the end because the writer does not know those offsets until it has written the data. Putting the map last lets a writer stream row groups out in one pass, remembering where each landed, and write the map when it is finished. Nothing is ever rewritten, which matters on object storage, where an object cannot be modified in place. The cost is paid by the reader, who must start at the end.

A file whose writer crashed before the footer was written has no map. Its data is all present and none of it can be read.

Building it#

The reader you called above is the book’s reader, which exists in Python (python/parquet_lab) and in Rust (crates/parquet-lab), and the tests hold the two to the same answers. This section builds the part of it the experiment used, in the order it ran. Each piece is quoted from the source, so what you read here is what ran above, and the tabs switch every excerpt on the page between the languages.

Little-endian integers#

The footer length is a little-endian unsigned integer: the first byte is the least significant. Reading one is a sum of each byte times a power of 256:

def read_le_u32(data: bytes) -> int:
    """Read four bytes as an unsigned little-endian integer: the first byte is the least
    significant.

    ``7c 01 00 00`` is ``0x7c + 0x01 × 256 = 380``.
    """
    return data[0] | data[1] << 8 | data[2] << 16 | data[3] << 24

pub fn read_le_u32(bytes: [u8; 4]) -> u32 {
    u32::from(bytes[0])
        | u32::from(bytes[1]) << 8
        | u32::from(bytes[2]) << 16
        | u32::from(bytes[3]) << 24
}

Parquet uses little-endian order everywhere it stores a fixed-width number: the footer length, and every integer and float in a page (ch05).

The trailer#

Parsing the trailer checks the magic first, and only then trusts the length:

def parse_trailer(last8: bytes, file_size: int) -> Trailer:
    """Decode the trailer of a file of ``file_size`` bytes from its last eight bytes."""
    if file_size < MIN_FILE_LEN:
        raise TooShort(file_size)
    span = Span(file_size - TRAILER_LEN, file_size)
    magic = last8[4:8]
    if magic == MAGIC_ENCRYPTED_FOOTER:
        raise EncryptedFooter()
    if magic != MAGIC:
        raise BadMagic(magic, Span(span.start + 4, span.end))
    footer_length = read_le_u32(last8[0:4])
    return Trailer(last8, span, footer_length, magic)

pub fn parse_trailer(last8: [u8; 8], file_size: u64) -> Result<Trailer, FormatError> {
    if file_size < MIN_FILE_LEN {
        return Err(FormatError::TooShort { file_size });
    }
    let span = Span::new(file_size - TRAILER_LEN, file_size);
    let magic = [last8[4], last8[5], last8[6], last8[7]];
    if magic == MAGIC_ENCRYPTED_FOOTER {
        return Err(FormatError::EncryptedFooter);
    }
    if magic != MAGIC {
        return Err(FormatError::BadMagic {
            found: magic,
            span: Span::new(span.start + 4, span.end),
        });
    }
    let footer_length = read_le_u32([last8[0], last8[1], last8[2], last8[3]]);
    Ok(Trailer {
        bytes: last8,
        span,
        footer_length,
        magic,
    })
}

PARE is the closing magic of a file whose footer is encrypted. The reader recognises it and says so, rather than misreading ciphertext as a footer length. ch13 reads those files.

Where the footer is#

The footer ends where the trailer starts, and starts footer_length bytes earlier. It cannot start inside the opening magic, so a length that would put it there means the file is damaged:

def footer_span(file_size: int, footer_length: int) -> Span:
    """Where the footer is: the ``footer_length`` bytes that end where the trailer begins.

    ::

        footer_start = file_size - 8 - footer_length
        footer_end   = file_size - 8

    The footer cannot overlap the opening magic, so a length that would put its start before
    byte 4 means the trailer is lying, and the reader stops rather than trusting it.
    """
    if file_size < MIN_FILE_LEN:
        raise TooShort(file_size)
    footer_end = file_size - TRAILER_LEN
    available = footer_end - len(MAGIC)
    if footer_length > available:
        raise FooterLengthTooLarge(footer_length, available)
    return Span(footer_end - footer_length, footer_end)

pub fn footer_span(file_size: u64, footer_length: u32) -> Result<Span, FormatError> {
    if file_size < MIN_FILE_LEN {
        return Err(FormatError::TooShort { file_size });
    }
    let footer_end = file_size - TRAILER_LEN;
    let available = footer_end - MAGIC.len() as u64;
    if u64::from(footer_length) > available {
        return Err(FormatError::FooterLengthTooLarge {
            footer_length,
            available,
        });
    }
    Ok(Span::new(footer_end - u64::from(footer_length), footer_end))
}

A span here is half-open: start is included and end is not. An HTTP range is inclusive at both ends. The object store converts one to the other in one place, a span’s http_range, because getting it wrong by one byte is the most common range-request bug there is.

The object store#

The reader asks for bytes through an interface: a protocol in Python, a trait in Rust. head and get are all this chapter needs; later chapters list a table’s files and group requests into phases. why is instrumentation: the reader says what it wants the bytes for, and the trace records it.

class ObjectStore(Protocol):
    """What a Parquet reader needs from storage.

    ``why`` is not part of any storage protocol. It is instrumentation: the reader says what it
    wants the bytes for, the trace records it, and the browser can then answer "why did the
    reader request these bytes?" from the reader's own words rather than from a caption.
    """

    def head(self, key: str, why: str) -> int: ...

    def get(self, key: str, range: GetRange, why: str) -> GetResult: ...

    def list(self, prefix: str, why: str) -> list[tuple[str, int]]:
        """The keys that start with ``prefix``, in order, with each object's size: S3's
        ``LIST`` (ch14)."""
        ...

    def next_phase(self) -> None:
        """Say that the next request depends on the answers to earlier ones, so it cannot start
        until they have finished. A store that does not model time ignores it."""

pub trait ObjectStore {
    fn head(&mut self, key: &str, why: &str) -> Result<u64, StoreError>;
    fn get(&mut self, key: &str, range: GetRange, why: &str) -> Result<GetResult, StoreError>;

    /// The keys that start with `prefix`, in order, with each object's size: S3's `LIST` (ch14).
    fn list(&mut self, prefix: &str, why: &str) -> Vec<(String, u64)>;

    /// Say that the next request depends on the answers to earlier ones, so it cannot start
    /// until they have finished. A store that does not model time ignores it.
    fn next_phase(&mut self) {}
}

MemoryStore serves ranges the way S3 does: a range that runs past the end of the object is cut short rather than refused, and a suffix range longer than the object returns all of it. TracingStore wraps any store, records each request, and charges it a simulated cost from a NetworkModel: a fixed latency per request, plus bytes over bandwidth. The model is simple on purpose. It captures the one property that shapes Parquet readers, that a request is expensive and a byte is cheap, and it gives the same numbers every time the book is built.

Opening a file#

The whole procedure, with both shortcuts:

def read_footer(store: ObjectStore, key: str, options: FooterOptions) -> FooterRead:
    """Open ``key`` in ``store``: find the footer, fetch it, and decode it."""
    want = max(options.prefetch, TRAILER_LEN)

    # Steps 1 and 2: the size, and the tail.
    match options.size:
        case Head():
            size = store.head(key, "learn the file size, to know where the end is")
            store.next_phase()
            tail = store.get(key, tail_range(size, want), "read the trailer")
        case Known(size):
            tail = store.get(key, tail_range(size, want), "read the trailer")
        case SuffixRange():
            tail = store.get(key, Suffix(want), "read the trailer; the response carries the file size")
    file_size = tail.object_size
    if len(tail.data) < TRAILER_LEN:
        raise TooShort(file_size)
    trailer = parse_trailer(tail.data[-TRAILER_LEN:], file_size)

    # Step 3: where the footer is.
    footer = footer_span(file_size, trailer.footer_length)

    # Step 4: fetch whatever part of the footer the tail did not already hold.
    footer_was_prefetched = tail.span.start <= footer.start
    if footer_was_prefetched:
        start = footer.start - tail.span.start
        footer_bytes = tail.data[start : start + footer.length]
    else:
        missing = Span(footer.start, tail.span.start)
        store.next_phase()
        head = store.get(key, Bounded(missing), "read the footer: the trailer says where it starts")
        overlap = footer.end - tail.span.start
        footer_bytes = head.data + tail.data[:overlap]

    metadata = decode_file_metadata(footer_bytes, footer.start)
    return FooterRead(file_size, tail.span, trailer, footer, footer_was_prefetched, footer_bytes, metadata)

pub fn read_footer<S: ObjectStore>(
    store: &mut S,
    key: &str,
    options: FooterOptions,
) -> Result<FooterRead, ReadError> {
    let want = options.prefetch.max(TRAILER_LEN);

    // Steps 1 and 2: the size, and the tail.
    let tail = match options.size {
        SizeSource::Head => {
            let size = store.head(key, "learn the file size, to know where the end is")?;
            store.next_phase();
            store.get(key, tail_range(size, want), "read the trailer")?
        }
        SizeSource::Known(size) => store.get(key, tail_range(size, want), "read the trailer")?,
        SizeSource::SuffixRange => store.get(
            key,
            GetRange::Suffix(want),
            "read the trailer; the response carries the file size",
        )?,
    };
    let file_size = tail.object_size;
    if tail.bytes.len() < TRAILER_LEN as usize {
        return Err(FormatError::TooShort { file_size }.into());
    }
    let t = &tail.bytes[tail.bytes.len() - TRAILER_LEN as usize..];
    let trailer = parse_trailer([t[0], t[1], t[2], t[3], t[4], t[5], t[6], t[7]], file_size)?;

    // Step 3: where the footer is.
    let footer = footer_span(file_size, trailer.footer_length)?;

    // Step 4: fetch whatever part of the footer the tail did not already hold.
    let footer_was_prefetched = tail.span.start <= footer.start;
    let footer_bytes = if footer_was_prefetched {
        let from = (footer.start - tail.span.start) as usize;
        tail.bytes[from..from + footer.len() as usize].to_vec()
    } else {
        let missing = Span::new(footer.start, tail.span.start);
        store.next_phase();
        let head = store.get(
            key,
            GetRange::Bounded(missing),
            "read the footer: the trailer says where it starts",
        )?;
        let mut bytes = head.bytes;
        let overlap = (footer.end - tail.span.start) as usize;
        bytes.extend_from_slice(&tail.bytes[..overlap]);
        bytes
    };

    let metadata = decode_file_metadata(&footer_bytes, footer.start)?;
    Ok(FooterRead {
        file_size,
        tail: tail.span,
        trailer,
        footer,
        footer_was_prefetched,
        footer_bytes,
        metadata,
    })
}

When the first read did not cover the footer, the reader requests only the part it is missing and joins it to the tail it already has. It never requests a byte twice.

Reading the footer#

The footer is serialised with Apache Thrift’s compact protocol. Every field starts with a header byte: the high four bits are the difference between this field’s id and the previous one, and the low four bits are its type.

        delta = byte >> 4
        nibble = byte & 0x0F
        wire = WireType.from_nibble(nibble)
        if wire is None:
            raise UnknownWireType(nibble, header_start)
        if delta == 0:
            id = _i16(r.read_zigzag())
        else:
            id = _i16(last_id + delta)
        last_id = id

        let delta = byte >> 4;
        let nibble = byte & 0x0f;
        let wire = WireType::from_nibble(nibble).ok_or(ThriftError::UnknownWireType {
            nibble,
            offset: header_start,
        })?;
        let id = if delta == 0 {
            r.read_zigzag()? as i16
        } else {
            last_id + i16::from(delta)
        };
        last_id = id;

Because every field names itself, the decoder can walk a footer without knowing Parquet’s schema, and a field it has never heard of still decodes. The parquet_thrift module then gives the fields their names. Select any byte of the footer in the panel above and the inspector reads it as a field header, so you can check the decoder by hand. ch03 reads the schema out of this structure.

Checking it#

The reader is tested against the fixtures, and the expected values come from pyarrow, not from the reader. When pyarrow wrote each fixture it also wrote a manifest of what it wrote: sizes, offsets, encodings and the footer length. The tests compare the reader with pyarrow:

python3 -m pytest python/tests

cargo test -p parquet-lab --test fixtures

A further test holds the Python and Rust readers to each other: every call a lab can make, on every fixture and on damaged copies of one, must return the same JSON from both.

python3 -m pytest tests/test_python.py

Ask a library#

You will rarely open a file with the book’s reader. At work you ask a library, and every library opens a file the way you have: it reads the trailer, then the footer, and hands you what the footer says. pyarrow’s read_metadata and the parquet crate’s ParquetMetaDataReader do that here:

# The same facts from pyarrow, which reads the trailer and parses the footer for you.
import pyarrow.parquet as pq

metadata = pq.read_metadata("fixtures/tiny.parquet")
print("footer length:", metadata.serialized_size)
print("rows:", metadata.num_rows, "in", metadata.num_row_groups, "row group")
for i in range(metadata.num_columns):
    chunk = metadata.row_group(0).column(i)
    # A chunk starts with its dictionary page when it has one, and its first data page otherwise.
    start = chunk.dictionary_page_offset or chunk.data_page_offset
    print(f"column chunk {chunk.path_in_schema}: bytes {start} to {start + chunk.total_compressed_size}")

//! The same facts from the `parquet` crate, which reads the trailer and parses the footer for you.

use parquet::file::metadata::{FooterTail, ParquetMetaDataReader};

fn main() {
    let file =
        std::fs::File::open("fixtures/tiny.parquet").expect("run this from the repository's root");
    let metadata = ParquetMetaDataReader::new()
        .parse_and_finish(&file)
        .expect("a Parquet file");

    // The crate checks the trailer as it parses; it also decodes one on its own, from eight bytes.
    let data = std::fs::read("fixtures/tiny.parquet").unwrap();
    let tail = FooterTail::try_from(&data[data.len() - 8..]).expect("a Parquet trailer");
    println!("footer length: {}", tail.metadata_length());
    println!(
        "rows: {} in {} row group",
        metadata.file_metadata().num_rows(),
        metadata.num_row_groups()
    );
    for chunk in metadata.row_group(0).columns() {
        // A chunk starts with its dictionary page when it has one, and its first data page otherwise.
        let (start, length) = chunk.byte_range();
        println!(
            "column chunk {}: bytes {start} to {}",
            chunk.column_path().string(),
            start + length
        );
    }
}

The footer length and the bytes of each column chunk are the ones you found by hand at the start of the chapter, and the ones in the table of regions. The test that runs these steps checks both against pyarrow’s manifest. From here on a chapter that builds something ends by asking a library the same question, so that you know where each fact lives in the tools you already use.

In Python, the first run loads pyarrow into the page, a much larger download than the other steps. The page’s pyarrow is an older version than the desk’s, for a reason Appendix A gives; for this step the two print the same. In Rust the step needs the parquet crate, which lives in walkthroughs/libraries, a workspace of its own so that the book’s reader keeps no dependencies. Run it in a Codespace, or at a desk:

cargo run -q --manifest-path walkthroughs/libraries/Cargo.toml --bin footer_with_a_library

What this cannot tell you#

How a real network behaves. The simulated store charges a fixed latency and a fixed bandwidth. Real object stores have variable latency, a long tail of slow requests, throttling, and connection setup costs. The simulation shows why readers minimise round trips; it cannot tell you how long your reads will take.

Whether a prefetch fits a real footer. The fixtures have footers smaller than a kilobyte. A footer grows with the number of row groups times the number of columns, and a wide table’s footer can be far larger than any sensible prefetch. No fixture in this book has a footer that large. ch11 shows how fast one grows with the number of row groups.

What lies between the data and the footer. Page indexes and Bloom filters, when a writer includes them, sit after the row groups and before the footer. The fixtures have neither, so here the footer follows the last column chunk directly. ch09 adds them.

What an encrypted file looks like. The reader stops at PARE. ch13 goes further.

Key takeaways#

Problems#

Four, in exercises/python/anatomy_of_a_parquet_file.py, or in Rust in exercises/src/anatomy_of_a_parquet_file.rs. The first three have tests that fail until you solve them. Each test compares your function with the book’s reader or with pyarrow, across many cases, so a hard-coded answer does not pass. The fourth has no test, and says why.

The editor below holds the Python stubs and keeps your edits in this browser. Run the tests runs the graders in the page, the same tests that run at a desk.

2.1 The footer length. Decode the footer length from the last eight bytes of a file, writing the little-endian arithmetic yourself.

2.2 Where the footer is. Given a file size and a footer length, return the footer’s byte range, or refuse when no valid file could have them.

2.3 How many requests. A reader that knows the file size reads some number of bytes from the end, then fetches whatever part of the footer it missed. Predict how many GET requests it makes. The test runs the traced reader and counts.

2.4 Your own files. No test: the files are yours, and nothing here can see them. Take a Parquet file your systems write. Read its last eight bytes (any hex viewer will do) and work out its footer length. Compare it with the prefetch size of the engine that reads it, which is usually in the engine’s documentation or configuration. A good answer states the file’s size, its footer length, the engine’s prefetch, and how many round trips you therefore expect the engine to make when it opens the file. If you can trace the engine’s requests, check your prediction against the trace. A disagreement means the engine does something this chapter has not described, and finding out what is the useful result.

For the Rust stubs, or to work at a desk, run the chapter’s graders with these commands. Add -k problem_2_1 in Python, or problem_2_1 before the -- in Rust, to run one problem:

python3 -m pytest exercises/python/tests/test_anatomy_of_a_parquet_file.py --problems

cargo test -p exercises --test anatomy_of_a_parquet_file -- --ignored

Where to go next#

• The file layout, magic and footer are specified in the Apache Parquet format specification, and every footer structure is defined in parquet.thrift.
• The Thrift compact protocol is specified in the Apache Thrift repository.
• Range requests, including suffix ranges, are defined in RFC 9110, section 14.
• ch03 reads the schema out of the footer you decoded here.

Built from commit c32f48e.