ch08Metadata and statistics

What you build: Sort orders for every type, and the rules for which statistics a reader may use.

The question#

What does the footer know that lets a reader avoid reading the data?

Every chapter so far read data to answer a question. The footer can answer some questions without it. Each column chunk’s metadata carries a summary of the chunk’s values: the smallest, the largest, and how many are null. A reader looking for orders above a certain amount can compare that amount with each chunk’s maximum and skip the chunks that cannot hold a match. ch09 does that.

First there is a narrower question. A minimum is stored as bytes, and bytes mean nothing without an order. When is the footer’s minimum the value a reader thinks it is?

The experiment#

Read the statistics yourself#

statistics.parquet holds twelve orders, four to a row group. Each column was chosen for a way statistics are misread. Appendix B says how pyarrow wrote it.

Each step below is a few lines of code. In Python, run them in the page, change them with Edit, and run them again. In Rust, open them in a Codespace, or run one at a desk with cargo run -p walkthroughs --bin and its name. Each step finds the footer as ch02 did, and decodes it with the reader’s decode_file_metadata from ch03.

Four bytes, two numbers. customer_id holds customer numbers, and some are above 2^31. The footer stores each row group’s smallest and largest in four bytes, least significant first. Read the bytes two ways:

# Take a column's bounds from the footer, and read their bytes two ways.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import metadata

data = open("fixtures/statistics.parquet", "rb").read()
length = int.from_bytes(data[-8:-4], "little")
base = len(data) - 8 - length
md = metadata.decode_file_metadata(data[base:-8], base)
column = 1  # customer_id; try 2, which is delta
e = md.leaves()[column]
print(e.name, e.physical_type.name, e.logical_type)

for g, row_group in enumerate(md.row_groups):
    s = row_group.columns[column].statistics
    low, high = s.min_value, s.max_value  # four bytes each, least significant first
    unsigned = [int.from_bytes(b, "little") for b in (low, high)]
    signed = [int.from_bytes(b, "little", signed=True) for b in (low, high)]
    print(f"row group {g}: max {high.hex(' ')}; unsigned {unsigned}, signed {signed}")

//! Take a column's bounds from the footer, and read their bytes two ways.

use parquet_lab::metadata::decode_file_metadata;

fn main() {
    let path = "fixtures/statistics.parquet";
    let data = std::fs::read(path).expect("run this from the repository's root");
    let n = data.len();
    let length = u32::from_le_bytes(data[n - 8..n - 4].try_into().unwrap()) as usize;
    let base = n - 8 - length;
    let md = decode_file_metadata(&data[base..n - 8], base as u64).expect("a footer");
    let column = 1; // customer_id; try 2, which is delta
    let e = md.leaves()[column];
    let kind = e.physical_type.expect("a leaf").name();
    let logical = e.logical_type.as_ref().expect("a logical type");
    println!("{} {kind} {logical}", e.name);

    for (g, row_group) in md.row_groups.iter().enumerate() {
        let chunk = &row_group.columns[column];
        let s = chunk.statistics.as_ref().expect("statistics");
        // Four bytes each, least significant first.
        let low: [u8; 4] = s.min_value.as_deref().unwrap().try_into().unwrap();
        let high: [u8; 4] = s.max_value.as_deref().unwrap().try_into().unwrap();
        let unsigned = [u32::from_le_bytes(low), u32::from_le_bytes(high)];
        let signed = [i32::from_le_bytes(low), i32::from_le_bytes(high)];
        print!("row group {g}: max {high:02x?}; ");
        println!("unsigned {unsigned:?}, signed {signed:?}");
    }
}

The schema says customer_id is an INT32 annotated as an unsigned integer. Read as unsigned, every row group’s maximum is a customer number above 2^31. Read as signed, the same four bytes are negative, and the maximum falls below the minimum. The minimums are small, so both readings agree on them. Nothing in the bytes says which reading is right. Only the annotation does.

Change the column to 2, which is delta, a signed 8-bit integer stored in an INT32. Now the signed reading is the right one, and the unsigned reading turns its negative minimums into numbers above 2^31.

Bytes in order. Strings are compared byte by byte, as UTF-8 writes them. Sort four of the fixture’s cities by their bytes, then by the same bytes read as signed numbers:

# Sort four cities by their UTF-8 bytes, then by the same bytes read as signed numbers.
cities = ["Zürich", "Łódź", "Aarhus", "Århus"]
for city in cities:
    print(city, city.encode().hex(" "))

def signed(city):  # each byte as Java's byte reads it: 0x80 and above are negative
    return [b - 256 if b > 127 else b for b in city.encode()]

print("unsigned bytes:", ", ".join(sorted(cities, key=str.encode)))
print("signed bytes:  ", ", ".join(sorted(cities, key=signed)))
print("Python's sort: ", ", ".join(sorted(cities)))  # by code point, as UTF-8's bytes are

//! Sort four cities by their UTF-8 bytes, then by the same bytes read as signed numbers.

fn main() {
    let mut cities = ["Zürich", "Łódź", "Aarhus", "Århus"];
    for city in cities {
        println!("{city} {:02x?}", city.as_bytes());
    }
    cities.sort(); // a str sorts by its bytes, each unsigned
    println!("unsigned bytes: {cities:?}");
    // Each byte as Java's byte reads it: 0x80 and above are negative.
    cities.sort_by_key(|city| city.bytes().map(|b| b as i8).collect::<Vec<_>>());
    println!("signed bytes:   {cities:?}");
}

UTF-8 writes each ASCII letter as one byte below 0x80, and every other letter as two or more bytes of 0x80 and above. So Århus and Łódź sort after Zürich by their bytes. Read each byte as signed, as Java does, and those bytes become negative: both names move in front of Aarhus. Python’s own sort of the strings agrees with the bytes, because UTF-8 keeps the order of code points. Rust sorts a str by its bytes.

Add "Écija" or "Ängelholm", two more of the fixture’s cities, and see where each order puts them.

The reader decides. The reader chooses an order for each column from its physical type and its annotation, then asks its bounds function whether it may use each column chunk’s minimum and maximum:

# Ask the reader which bounds it may use, column chunk by column chunk, and why not.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import encoding, logical, metadata, schema, stats

data = open("fixtures/statistics.parquet", "rb").read()
length = int.from_bytes(data[-8:-4], "little")
base = len(data) - 8 - length
md = metadata.decode_file_metadata(data[base:-8], base)
old_writer = False  # try True: keep only the deprecated min and max, as old writers did

def show(leaf, v):  # through the logical type if it has one, else the physical type
    t, lt = leaf.physical_type, leaf.logical_type
    return (lt and logical.interpret(t, lt, v)) or encoding.plain_scalar(t, v)

for leaf in schema.leaves(schema.build(md.schema)):
    comparator = stats.Comparator.for_leaf(leaf, md.schema[leaf.element].converted_type)
    type_order = md.column_orders[leaf.column] == "TYPE_ORDER"
    print(f"{leaf.dotted_path()}, compared as {comparator.description}:")
    for g, row_group in enumerate(md.row_groups):
        s = row_group.columns[leaf.column].statistics
        if s is None:
            print(f"  row group {g}: no statistics")
            continue
        if old_writer:
            s.min_value = s.max_value = None
        try:
            b = stats.bounds(s, comparator, type_order)
            low, high = show(leaf, b.min), show(leaf, b.max)
            print(f"  row group {g}: {low} to {high}, from {b.source}")
        except stats.Unusable as refused:
            print(f"  row group {g}: refused, {refused}; null count {s.null_count}")

//! Ask the reader which bounds it may use, column chunk by column chunk, and why not.

use parquet_lab::encoding::plain_scalar;
use parquet_lab::logical::interpret;
use parquet_lab::metadata::decode_file_metadata;
use parquet_lab::schema::{build, leaves, Leaf};
use parquet_lab::stats::{bounds, Comparator};

/// Through the logical type if it has one, else the physical type.
fn show(leaf: &Leaf, v: &[u8]) -> String {
    let t = leaf.physical_type;
    let read = leaf.logical_type.as_ref().and_then(|l| interpret(t, l, v));
    read.or_else(|| plain_scalar(t, v)).unwrap_or_default()
}

fn main() {
    let path = "fixtures/statistics.parquet";
    let data = std::fs::read(path).expect("run this from the repository's root");
    let n = data.len();
    let length = u32::from_le_bytes(data[n - 8..n - 4].try_into().unwrap()) as usize;
    let base = n - 8 - length;
    let md = decode_file_metadata(&data[base..n - 8], base as u64).expect("a footer");
    // Try true: keep only the deprecated min and max, as old writers did.
    let old_writer = false;

    for leaf in leaves(&build(&md.schema).expect("a schema")) {
        let converted = md.schema[leaf.element].converted_type.as_deref();
        let comparator = Comparator::for_leaf(&leaf, converted);
        let type_order = md.column_orders.as_ref().unwrap()[leaf.column] == "TYPE_ORDER";
        println!("{}, compared as {}:", leaf.dotted_path(), comparator.name());
        for (g, row_group) in md.row_groups.iter().enumerate() {
            let Some(mut s) = row_group.columns[leaf.column].statistics.clone() else {
                println!("  row group {g}: no statistics");
                continue;
            };
            if old_writer {
                (s.min_value, s.max_value) = (None, None);
            }
            match bounds(&s, comparator, type_order) {
                Ok(b) => {
                    let (min, max) = (show(&leaf, &b.min), show(&leaf, &b.max));
                    println!("  row group {g}: {min} to {max}, from {:?}", b.source);
                }
                Err(refused) => {
                    let nulls = s.null_count.unwrap_or(0);
                    println!("  row group {g}: refused, {refused}; null count {nulls}");
                }
            }
        }
    }
}

Every column chunk with a minimum and maximum gives a range in its column’s order. customer_id’s maximums are the unsigned numbers, and city’s maximum in row group 0 is Łódź, as the second step predicted. order_id’s ranges do not overlap, because the rows were written in order: a reader looking for one order reads one row group. temp_c holds a NaN in row group 0, and the bounds leave it out. Its minimum in row group 1 is -0.

Two kinds of chunk give nothing. coupon in row group 1 has no minimum and no maximum, and its null count says why: every value is null. note has no statistics at all, because the writer was told not to keep them.

Set old_writer to true (True in Python). The step then deletes min_value and max_value, and keeps only the deprecated min and max, as a writer from before the newer fields would have. The reader still accepts order_id, delta and temp_c. customer_id and the strings have nothing left, because pyarrow wrote no deprecated fields for them. The reader refuses amount, although pyarrow wrote both pairs for it, and the section on the deprecated fields below says why. Then set old_writer back, and make type_order false, as if the footer had no column_orders: every chunk with statistics is refused.

What the footer records#

This is which fields of the Statistics structure the writer set for each column of the first row group:

• min_value and max_value are the bounds in the column’s own sort order.
• min and max are older fields with the same purpose. They are deprecated. pyarrow still writes them, as copies of min_value and max_value, for every column whose order is signed. Here that is every column but the unsigned integers and the strings, as the table shows.
• null_count counts the nulls. A chunk whose null count equals its value count holds no values at all.
• is_min_value_exact and is_max_value_exact say whether each bound is a value in the chunk. A writer may shorten a long string to keep the footer small, and the shortened bound is then only a bound.
• distinct_count is defined and rarely written. pyarrow does not write it.

Sort orders#

The same bytes sort differently depending on what they hold. The format fixes an order for every type:

The order of strings is the order of their bytes in UTF-8, as the second step showed. It is not alphabetical order for a person: Århus sorts after Zürich.

What a mistaken order gives#

Each column in the fixture has a likely mistake. The reader decodes the values, finds the smallest and largest in the column’s order, and checks them against the footer. Then it finds them again in the mistaken order:

Every mistaken result is confidently wrong, and the damage comes when two orders meet: bounds computed in one and compared in the other. A reader comparing strings correctly against the mistaken city bounds finds Aarhus below their minimum, and skips the row group that holds it. One comparing customer numbers correctly against the mistaken customer_id bounds skips the row group holding the largest of them. Nothing in the bytes warns of it; only the column’s type does.

The deprecated fields#

The first versions of the format defined min and max without saying in what order. The most widely used writer of the time compared everything as signed values, including strings, whose bytes Java treats as signed. Its string statistics were wrong for any value with a byte above 0x7f, and readers could not tell which files were affected.

The fix added min_value and max_value, and a column_orders list in the footer that says, column by column, which order they use. TYPE_ORDER is the order in the table above, and the one every fixture’s footer names; this reader treats any other as unknown. A reader’s rules follow from that history:

• Use min_value and max_value when the footer has column_orders. Without it, their order is undefined.
• Use min and max only where signed comparison is the right order and the values are not byte arrays. That rules out amount, a decimal stored in bytes. pyarrow’s copies for it are right, but a reader cannot tell pyarrow’s from the old writer’s, which compared every byte as signed. For a decimal’s bytes after the first, that order is wrong.
• Never use a bound that is NaN.

Some readers go further, and ignore statistics from particular releases of particular writers, which they recognise by the footer’s created_by string. This reader does not.

Floating point#

Floats have two traps. NaN is not less than, equal to or greater than anything, so a writer leaves it out of the bounds, and a reader looking for NaN cannot use them. And -0 equals +0, though their bytes differ. The format asks writers to store a zero minimum as -0 and a zero maximum as +0, and asks readers to remember that a chunk with either may hold both. The -0 the third step printed for temp_c in row group 1 is the first rule at work.

What else the footer says#

Beyond statistics, each row group records its row count and size, and may list the columns its rows are sorted by. sorting_columns is a claim: a reader may rely on it and cannot check it without reading every value. Each column chunk may also carry size statistics: the number of bytes its strings take once decoded, and a histogram of its definition and repetition levels, from which a reader can count nulls and list elements without reading a page. pyarrow’s metadata does not show them; the parquet crate’s does, as unencoded_byte_array_data_bytes and the level histograms.

All of it costs bytes, in a place every reader must fetch:

In a small file the footer can outweigh the data. A table with hundreds of columns and many row groups can have a footer of megabytes, and every query pays for reading and decoding it before it reads anything else. ch11 shows a footer growing with the number of row groups.

Building it#

The steps read bounds out of the footer, compared bytes in two orders, and asked the reader which bounds it may use. This section builds the two parts of the reader the last step called: the comparator for a column, and the rules in bounds. The tabs switch every excerpt on the page between the two languages.

The comparator for a column#

The order comes from the physical type and the annotation together:

    def for_leaf(leaf: Leaf, converted: str | None) -> Comparator:
        """The comparator for a leaf column, from its physical type and its annotation."""
        lt = leaf.logical_type
        unsigned = (lt is not None and lt.name == "INTEGER" and lt.signed is False) or converted in (
            "UINT_8",
            "UINT_16",
            "UINT_32",
            "UINT_64",
        )
        decimal = (lt is not None and lt.name == "DECIMAL") or converted == "DECIMAL"
        t = leaf.physical_type
        if t == 0:
            return Comparator.BOOL
        if t == 1:
            return Comparator.U32 if unsigned else Comparator.I32
        if t == 2:
            return Comparator.U64 if unsigned else Comparator.I64
        if t == 4:
            return Comparator.F32
        if t == 5:
            return Comparator.F64
        if t == 7 and lt is not None and lt.name == "FLOAT16":
            return Comparator.F16
        if t == 7 and converted == "INTERVAL":
            return Comparator.NONE
        if t in (6, 7):
            return Comparator.BIG_ENDIAN_SIGNED if decimal else Comparator.BYTES
        return Comparator.NONE  # INT96

    pub fn for_leaf(leaf: &Leaf, converted: Option<&str>) -> Comparator {
        let logical = leaf.logical_type.as_ref();
        let unsigned = matches!(logical, Some(LogicalType::Integer { signed: false, .. }))
            || matches!(
                converted,
                Some("UINT_8" | "UINT_16" | "UINT_32" | "UINT_64")
            );
        let decimal =
            matches!(logical, Some(LogicalType::Decimal { .. })) || converted == Some("DECIMAL");
        match leaf.physical_type.0 {
            0 => Comparator::Bool,
            1 if unsigned => Comparator::U32,
            1 => Comparator::I32,
            2 if unsigned => Comparator::U64,
            2 => Comparator::I64,
            4 => Comparator::F32,
            5 => Comparator::F64,
            7 if matches!(logical, Some(LogicalType::Float16)) => Comparator::F16,
            7 if converted == Some("INTERVAL") => Comparator::None,
            6 | 7 if decimal => Comparator::BigEndianSigned,
            6 | 7 => Comparator::Bytes,
            _ => Comparator::None, // INT96
        }
    }

The rules#

bounds applies the rules in the order the chapter gives them, and says why when it refuses. It refuses a type without an order first, then prefers min_value and max_value, which it uses only when the footer names their order and neither is NaN:

def bounds(stats: Statistics, comparator: Comparator, type_order: bool) -> Bounds:
    """The bounds a reader may use, or why it may not use any.

    ``type_order`` is whether the footer's ``column_orders`` gives this column ``TYPE_ORDER``.
    """

    def is_nan(v: bytes) -> bool:
        return comparator.compare(v, v) is None

    if comparator.order() == UNDEFINED:
        raise Unusable("this type has no defined sort order, so its minimum and maximum mean nothing")
    if stats.min_value is not None and stats.max_value is not None:
        if not type_order:
            raise Unusable(
                "the footer has no column_orders, so the order of min_value and max_value is undefined"
            )
        if is_nan(stats.min_value) or is_nan(stats.max_value):
            raise Unusable("a bound is NaN, which writers were never meant to store; ignore both")
        return Bounds(
            stats.min_value,
            stats.max_value,
            MIN_MAX_VALUE,
            stats.is_min_value_exact is not False,
            stats.is_max_value_exact is not False,
        )

pub fn bounds(
    stats: &Statistics,
    comparator: Comparator,
    type_order: bool,
) -> Result<Bounds, String> {
    let is_nan = |v: &[u8]| comparator.compare(v, v).is_none();
    if comparator.order() == SortOrder::Undefined {
        return Err(
            "this type has no defined sort order, so its minimum and maximum mean nothing".into(),
        );
    }
    if let (Some(min), Some(max)) = (&stats.min_value, &stats.max_value) {
        if !type_order {
            return Err(
                "the footer has no column_orders, so the order of min_value and \
                 max_value is undefined"
                    .into(),
            );
        }
        if is_nan(min) || is_nan(max) {
            return Err(
                "a bound is NaN, which writers were never meant to store; ignore both".into(),
            );
        }
        return Ok(Bounds {
            min: min.clone(),
            max: max.clone(),
            source: Source::MinMaxValue,
            min_exact: stats.is_min_value_exact.unwrap_or(true),
            max_exact: stats.is_max_value_exact.unwrap_or(true),
        });
    }

Without the newer fields, it falls back to the deprecated pair, but only for a column whose order is signed and whose values are not byte arrays. This is the branch that refused amount when you set old_writer:

    if stats.min is not None and stats.max is not None:
        # The deprecated fields were computed with signed comparison. That is right only for
        # types whose order is signed and that are not compared as bytes.
        byte_array = comparator in (Comparator.BYTES, Comparator.BIG_ENDIAN_SIGNED)
        if comparator.order() != SIGNED or byte_array:
            raise Unusable(
                "only the deprecated min and max are present, and they were computed with signed "
                f"comparison; this column's order is {comparator.description}"
            )
        if is_nan(stats.min) or is_nan(stats.max):
            raise Unusable("a bound is NaN, which writers were never meant to store; ignore both")
        return Bounds(stats.min, stats.max, DEPRECATED, True, True)
    raise Unusable("the footer records no minimum and maximum for this column chunk")

    if let (Some(min), Some(max)) = (&stats.min, &stats.max) {
        // The deprecated fields were computed with signed comparison. That is right only for
        // types whose order is signed and that are not compared as bytes.
        let byte_array = matches!(comparator, Comparator::Bytes | Comparator::BigEndianSigned);
        if comparator.order() != SortOrder::Signed || byte_array {
            return Err(format!(
                "only the deprecated min and max are present, and they were computed with signed \
                 comparison; this column's order is {}",
                comparator.name()
            ));
        }
        if is_nan(min) || is_nan(max) {
            return Err(
                "a bound is NaN, which writers were never meant to store; ignore both".into(),
            );
        }
        return Ok(Bounds {
            min: min.clone(),
            max: max.clone(),
            source: Source::Deprecated,
            min_exact: true,
            max_exact: true,
        });
    }
    Err("the footer records no minimum and maximum for this column chunk".into())
}

ch03’s reader used min and max whenever min_value and max_value were missing. That was the mistake this chapter describes, and the statistics decoder now keeps the two pairs apart.

Checking it#

For every column chunk of every fixture, the reader decodes the values, finds their minimum and maximum in its chosen order, and compares them with the bounds pyarrow wrote. They agree. The statistics fixture also shows that each mistaken order gets at least one of its columns wrong:

python3 -m pytest python/tests

cargo test -p parquet-lab --test fixtures

Ask a library#

pyarrow and the parquet crate both read the statistics for you. Ask each for row group 0:

# The same facts from pyarrow: bounds read through the logical type, and raw.
import pyarrow.parquet as pq

md = pq.read_metadata("fixtures/statistics.parquet")
row_group = md.row_group(0)  # try 1
print(f"the footer: {md.serialized_size} bytes; sorted by {row_group.sorting_columns}")
for i in range(md.num_columns):
    c = row_group.column(i)
    s, name = c.statistics, c.path_in_schema
    if s is None:  # the writer kept none
        print(f"{name}: no statistics")
    elif not s.has_min_max:
        print(f"{name}: no minimum or maximum, null count {s.null_count}")
    else:
        print(f"{name}: {s.min!r} to {s.max!r}, null count {s.null_count}")
        if s.max_raw != s.max:  # as the physical type reads them, where that differs
            print(f"  raw, as {s.physical_type}: {s.min_raw!r} to {s.max_raw!r}")

//! The same facts from the `parquet` crate: each column's sort order, and its statistics, which
//! it reads as the physical type says.

use parquet::file::metadata::ParquetMetaDataReader;
use parquet::file::statistics::Statistics;

fn main() {
    let path = "fixtures/statistics.parquet";
    let file = std::fs::File::open(path).expect("run this from the repository's root");
    let md = ParquetMetaDataReader::new()
        .parse_and_finish(&file)
        .expect("a Parquet file");
    let row_group = md.row_group(0); // try 1
    println!("sorted by {:?}", row_group.sorting_columns());
    for (i, chunk) in row_group.columns().iter().enumerate() {
        let order = md.file_metadata().column_order(i);
        print!("{}: {order}", chunk.column_path().string());
        match chunk.statistics() {
            Some(s) => {
                let (exact, deprecated) = (s.min_is_exact(), s.is_min_max_deprecated());
                println!(", exact {exact}, from the deprecated fields {deprecated}");
            }
            None => println!(", no statistics"),
        }
    }
    // The column order says unsigned; the crate hands you an i32 all the same.
    if let Some(Statistics::Int32(s)) = row_group.column(1).statistics() {
        println!("customer_id: {:?} to {:?}", s.min_opt(), s.max_opt());
    }
}

pyarrow reads the bounds through the logical type. Its min and max give customer_id’s maximum as the unsigned customer number, and city’s as a string. Its min_raw and max_raw give the physical type’s reading, where customer_id’s maximum is negative, as the first step’s signed reading was. It reports the footer’s size, which is the footer bytes in the cost table, and the row group’s sorting_columns. It has no attribute for the exact flags, and none for the column orders.

The crate reports each column’s order from column_orders: customer_id’s is TYPE_DEFINED_ORDER(UNSIGNED). Yet its statistics for an INT32 column hold an i32, and max_opt returns customer_id’s maximum as a negative number. The crate hands you the first step’s mistake, and applying the column order is left to you. It also says every bound is exact, and that none came from the deprecated fields, although pyarrow wrote those too: where both pairs are present, the crate reads min_value and max_value.

In Python, the first run loads pyarrow into the page, a much larger download than the other steps. In Rust the step needs the parquet crate, in walkthroughs/libraries, so it runs in a Codespace or at a desk:

cargo run -q --manifest-path walkthroughs/libraries/Cargo.toml --bin statistics_with_a_library

What this cannot tell you#

Whether a query can skip a chunk. The bounds say what a chunk may hold. Turning a query’s condition into a decision is ch09’s subject, along with the finer statistics kept for each page.

What shortened bounds look like. pyarrow wrote exact bounds for every value here. Writers that shorten long strings mark the bound as inexact, and the reader reports the flag, but no fixture exercises it.

Which writers wrote wrong statistics. The reader applies the format’s rules. Files from old writers can break those rules in ways only a list of writer versions can catch.

How people sort text. Byte order is not a collation. A query that compares strings the way a language does cannot use these bounds directly.

Key takeaways#

Problems#

Three, in exercises/python/metadata_and_statistics.py, or in Rust in exercises/src/metadata_and_statistics.rs. The first two have tests. The third has none.

The editor below holds the Python stubs and keeps your edits in this browser. Run the tests runs the graders in the page, the same tests that run at a desk.

8.1 Sort orders. Compare two values of a column in its sort order, for six kinds of column. The test finds each column chunk’s minimum and maximum with your order and checks them against the bounds pyarrow wrote.

8.2 Which bounds. Decide which bounds a reader may use. The test presents each chunk’s statistics as written, without column_orders, as an old writer would have set them, and with a NaN.

8.3 Your own footers. No test: the files are yours. Run PYTHONPATH=python python3 -m parquet_lab statistics FILE 0 COLUMN or cargo run -p pqlab -- statistics FILE 0 COLUMN on a file your systems write, for a few columns. Record which columns have statistics, whether the reader accepts them, and what share of the file the footer takes. Then find one column where the statistics could not help a query you run, and say why: no statistics, a range as wide as the column’s, or a comparison the order does not support. A good answer says what the writer could change.

For the Rust stubs, or to work at a desk, run the chapter’s graders with these commands. Add -k problem_8_1 in Python, or problem_8_1 before the -- in Rust, to run one problem:

python3 -m pytest exercises/python/tests/test_metadata_and_statistics.py --problems

cargo test -p exercises --test metadata_and_statistics -- --ignored

Where to go next#

• The Statistics and ColumnOrder structures, and the rules for floating point, are documented in parquet.thrift.
• Each logical type’s sort order is in LogicalTypes.md.
• The history of the signed comparison of strings is in PARQUET-686.
• ch09 uses these bounds, and the finer ones kept per page, to skip data.

Built from commit c32f48e.