ch03The type system

What you build: The schema tree rebuilt from the footer, with maximum levels, and logical types applied to values.

The question#

How do a handful of storage types carry strings, dates, decimals and timestamps?

ch02 found the footer and decoded it into numbered fields. One of those fields is the schema: the names and types of the columns. This chapter reads it. It answers two questions a reader must settle before it can decode a single value: which columns the file has, and what the bytes of each column mean.

The answer to the second is in two layers. A small set of physical types says how many bytes a value takes and in what order. A logical type, written beside the physical type in the schema, says what those bytes mean to a person. The same four bytes can be a count, a day, or an unsigned number, and only the schema says which.

The experiment#

Read the schema yourself#

types.parquet holds three orders, with a column for each physical type and the common logical types on top of them. Appendix B says how pyarrow wrote it. Each step below is a few lines of code. In Python, run them in the page, change them with Edit, and run them again. In Rust, open them in a Codespace, or run one at a desk with cargo run -p walkthroughs --bin and its name. Each step finds the footer as ch02 did, and hands its bytes to the book’s reader, which decodes the Thrift fields the way ch02 built it to.

The schema as stored. The footer’s second field is the schema. Print every element of it:

# The schema as the footer stores it: a flat list, one element per field, groups included.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import metadata

data = open("fixtures/types.parquet", "rb").read()
length = int.from_bytes(data[-8:-4], "little")
base = len(data) - 8 - length  # where the footer starts, as in ch02
md = metadata.decode_file_metadata(data[base:-8], base)

for e in md.schema:
    if e.num_children:
        print(e.name, e.num_children, "children", e.repetition)
    else:
        print(e.name, e.repetition, e.physical_type.name, e.logical_type or "")

//! The schema as the footer stores it: a flat list, one element per field, groups included.

use parquet_lab::metadata::decode_file_metadata;

fn main() {
    let data =
        std::fs::read("fixtures/types.parquet").expect("run this from the repository's root");
    let n = data.len();
    let length = u32::from_le_bytes(data[n - 8..n - 4].try_into().unwrap()) as usize;
    let base = n - 8 - length; // where the footer starts, as in ch02
    let md = decode_file_metadata(&data[base..n - 8], base as u64).expect("a footer");

    for e in &md.schema {
        let repetition = e.repetition.as_deref().unwrap_or("");
        match (e.num_children.unwrap_or(0), e.physical_type) {
            (0, Some(physical)) => {
                let logical = e.logical_type.as_ref().map(|l| l.to_string());
                let logical = logical.unwrap_or_default();
                println!("{} {repetition} {} {logical}", e.name, physical.name());
            }
            (children, _) => println!("{} {children} children {repetition}", e.name),
        }
    }
}

One line per element, in the footer’s order. The first is the root, schema. It has children and no type, and it names the whole record. Most of the others have a physical type and no children: they are leaves, and each leaf is one column of the file. The leaves come in column order, and the column chunks in every row group follow the same order.

shipping is neither a leaf nor the root. It has children and no type: it is a group, and the elements after it are its children. Nothing in the list says where a group ends. Its count of children is the only clue. Change the path to fixtures/nested.parquet and run the step again: a list is a group too, with a REPEATED field inside it.

The tree, rebuilt. A count of children is enough to rebuild the tree. Take an element; if it has children, take that many subtrees after it, each the same way:

# Rebuild the tree from the flat list: each group's count of children is all it takes.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import metadata, schema

data = bytearray(open("fixtures/types.parquet", "rb").read())
data[674] = 0x14  # the root's count of children, 10, as a zigzag varint; try 0x12, which is 9
length = int.from_bytes(data[-8:-4], "little")
base = len(data) - 8 - length
md = metadata.decode_file_metadata(bytes(data[base:-8]), base)

def walk(elements, depth=0, definition=0, repetition=0):
    e = next(elements)
    if depth:  # every optional or repeated field on the way down adds a level
        definition += e.repetition != "REQUIRED"
        repetition += e.repetition == "REPEATED"
    levels = f"  max levels: definition {definition}, repetition {repetition}"
    print("  " * depth + e.name + ("" if e.num_children else levels))
    for _ in range(e.num_children or 0):
        walk(elements, depth + 1, definition, repetition)

walk(iter(md.schema))
try:
    print("the reader's build finds", len(schema.leaves(schema.build(md.schema))), "columns")
except schema.SchemaError as refused:
    print("the reader refuses:", refused)

//! Rebuild the tree from the flat list: each group's count of children is all it takes.

use parquet_lab::metadata::{decode_file_metadata, SchemaElement};
use parquet_lab::schema;

fn main() {
    let mut data =
        std::fs::read("fixtures/types.parquet").expect("run this from the repository's root");
    data[674] = 0x14; // the root's count of children, 10, as a zigzag varint; try 0x12, which is 9
    let n = data.len();
    let length = u32::from_le_bytes(data[n - 8..n - 4].try_into().unwrap()) as usize;
    let base = n - 8 - length;
    let md = decode_file_metadata(&data[base..n - 8], base as u64).expect("a footer");

    walk(&mut md.schema.iter(), 0, 0, 0);
    match schema::build(&md.schema) {
        Ok(root) => {
            let columns = schema::leaves(&root).len();
            println!("the reader's build finds {columns} columns");
        }
        Err(refused) => println!("the reader refuses: {refused}"),
    }
}

/// Take one element and print it, then as many subtrees as it says it has children.
fn walk<'a>(
    elements: &mut impl Iterator<Item = &'a SchemaElement>,
    depth: usize,
    mut definition: u32,
    mut repetition: u32,
) {
    let e = elements.next().expect("an element for every child");
    if depth > 0 {
        // Every optional or repeated field on the way down adds a level.
        definition += u32::from(e.repetition.as_deref() != Some("REQUIRED"));
        repetition += u32::from(e.repetition.as_deref() == Some("REPEATED"));
    }
    let children = e.num_children.unwrap_or(0);
    let levels = format!("  max levels: definition {definition}, repetition {repetition}");
    let levels = if children > 0 { "" } else { &levels };
    println!("{}{}{levels}", "  ".repeat(depth), e.name);
    for _ in 0..children {
        walk(elements, depth + 1, definition, repetition);
    }
}

The walk consumes the list from the front, and the indentation is its depth. On the way down it counts the fields that may be absent. country is optional, so each of its values needs a level saying whether it is there. shipping.city is optional inside an optional group, so it needs a level that can say which of the two is missing. These counts are each leaf’s maximum levels, and ch04 needs them to decode the values. The book’s reader then rebuilds the same tree with schema.build, and finds the same columns.

Now damage the list. The step writes the root’s count of children back to the byte that holds it, as the zigzag varint 0x14. Change it to 0x12, one child fewer, and run the step again. The walk prints a shorter tree without complaint: it never looks at the elements after the root’s last child. shipping and its children now belong to no one, and the reader’s build refuses the file for that reason.

The same bytes, read two ways. Each column chunk’s statistics hold its smallest value, as bytes. Read three of them, first as the physical type stores them and then as the logical type says:

# The same bytes, read two ways: as the physical type stores them, then as the logical type says.
import sys
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import metadata

data = open("fixtures/types.parquet", "rb").read()
length = int.from_bytes(data[-8:-4], "little")
base = len(data) - 8 - length
md = metadata.decode_file_metadata(data[base:-8], base)
least = {c.dotted_path(): c.statistics.min_value for c in md.row_groups[0].columns}

raw = least["order_date"]  # INT32, DATE: days since 1970-01-01
days = int.from_bytes(raw, "little", signed=True)
print("order_date", raw.hex(" "), "->", days, "->", date(1970, 1, 1) + timedelta(days=days))
raw = least["paid_at"]  # INT64, TIMESTAMP(MICROS, UTC): microseconds since 1970-01-01, UTC
micros = int.from_bytes(raw, "little", signed=True)
epoch = datetime(1970, 1, 1, tzinfo=timezone.utc)
print("paid_at", raw.hex(" "), "->", micros, "->", epoch + timedelta(microseconds=micros))
raw = least["amount"]  # FIXED_LEN_BYTE_ARRAY(4), DECIMAL(9, 2): an integer of hundredths
cents = int.from_bytes(raw, "big", signed=True)  # most significant byte first
print("amount", raw.hex(" "), "->", cents, "->", Decimal(cents).scaleb(-2))

//! The same bytes, read two ways: as the physical type stores them, then as the logical type says.

use parquet_lab::logical::{date_from_days, decimal, timestamp, TimeUnit};
use parquet_lab::metadata::decode_file_metadata;

fn main() {
    let data =
        std::fs::read("fixtures/types.parquet").expect("run this from the repository's root");
    let n = data.len();
    let length = u32::from_le_bytes(data[n - 8..n - 4].try_into().unwrap()) as usize;
    let base = n - 8 - length;
    let md = decode_file_metadata(&data[base..n - 8], base as u64).expect("a footer");
    let chunks = &md.row_groups[0].columns;
    let least = |path: &str| {
        let chunk = chunks
            .iter()
            .find(|c| c.dotted_path() == path)
            .expect("a column");
        let stats = chunk.statistics.as_ref().expect("statistics");
        stats.min_value.clone().expect("a minimum")
    };

    // Rust's standard library has no calendar, so the dates come from the book's reader.
    let raw = least("order_date"); // INT32, DATE: days since 1970-01-01
    let days = i32::from_le_bytes(raw[..].try_into().unwrap());
    let date = date_from_days(i64::from(days));
    println!("order_date {} -> {days} -> {date}", hex(&raw));
    let raw = least("paid_at"); // INT64, TIMESTAMP(MICROS, UTC): microseconds since 1970-01-01, UTC
    let micros = i64::from_le_bytes(raw[..].try_into().unwrap());
    let at = timestamp(micros, TimeUnit::Micros, true);
    println!("paid_at {} -> {micros} -> {at}", hex(&raw));
    let raw = least("amount"); // FIXED_LEN_BYTE_ARRAY(4), DECIMAL(9, 2): an integer of hundredths
    let cents = i32::from_be_bytes(raw[..].try_into().unwrap()); // most significant byte first
    let amount = decimal(i128::from(cents), 2);
    println!("amount {} -> {cents} -> {amount}", hex(&raw));
}

/// Bytes as two hex digits each, spaced, as Python's `bytes.hex(" ")` writes them.
fn hex(bytes: &[u8]) -> String {
    let digits: Vec<String> = bytes.iter().map(|b| format!("{b:02x}")).collect();
    digits.join(" ")
}

order_date’s bytes are a little-endian INT32, and as a number they mean little. Its logical type, DATE, says the number counts days since 1970-01-01, and then it is a date. paid_at is the same idea at a finer grain: an INT64 count of microseconds. amount is the odd one out. Its integer is read most significant byte first, which is big-endian, and it counts hundredths. Change "big" to "little" (from_be_bytes to from_le_bytes in Rust), and the same bytes are a number nobody meant. Only the schema says which reading is right.

Rebuilt as a tree#

The tree you rebuilt is the schema. Parquet tools print it in a textual notation that comes from the Dremel paper:

message schema {
  required boolean is_paid;
  required int32 quantity (INTEGER(8, signed));
  required int32 store_id (INTEGER(16, unsigned));
  required int64 order_id;
  required float weight_kg;
  required fixed_len_byte_array(4) amount (DECIMAL(9, 2));
  optional binary country (STRING);
  required int32 order_date (DATE);
  required int64 paid_at (TIMESTAMP(MICROS, UTC));
  optional group shipping {
    optional binary city (STRING);
    optional binary postcode (STRING);
  }
}

Each line has three parts: a repetition, a type, and a name. The logical type, when there is one, follows in brackets.

The repetition says how many values the field has in each record:

• required: exactly one. It is never null.
• optional: zero or one. It may be null.
• repeated: zero or more. The field is a list.

A leaf’s path is the names from the root’s child down to it: shipping.city. A column chunk records its path, and the path is how a query names a nested column.

Every column, read two ways#

A physical type is a storage format. There are eight:

A logical type says what a stored value means. Here is the minimum of every column in types.parquet, as the footer stores it, read by its physical type alone, and read through its logical type:

Read down the last two columns. Where there is no logical type, the physical reading is the value. Where there is one, the physical reading is often a number nobody meant:

• order_date is an INT32 with the logical type DATE: a count of days since 1970-01-01.
• paid_at is an INT64 with TIMESTAMP: a count of microseconds since the same moment. The logical type also records the unit, and whether the value is an instant (UTC) or a wall-clock reading with no zone (local). The two look identical in the bytes and differ in meaning.
• amount is a DECIMAL: an integer to be divided by a power of ten, here with two digits after the point. pyarrow stored it in a FIXED_LEN_BYTE_ARRAY, and there the integer is big-endian. A decimal stored as bytes is big-endian in a BYTE_ARRAY too, and so is a UUID’s sixteen bytes. Every other number Parquet stores is little-endian.
• quantity and store_id are both INT32. The logical type says one is an 8-bit signed integer and the other a 16-bit unsigned one. Both ranges fit in a signed 32-bit integer, so here the two readings agree. They part for unsigned 32-bit and 64-bit integers, stored in INT32 and INT64: a reader that ignored the logical type would read a large value as negative.
• country is a BYTE_ARRAY with STRING: the bytes are UTF-8 text. Without the annotation they are opaque bytes, and a reader should not assume otherwise.

Encoders and decoders only ever handle the eight physical types. Every logical type is an interpretation applied after decoding. That keeps the part of a reader that touches bytes small, and lets the format add logical types without changing how anything is stored.

Older annotations, and field ids#

Files written before logical types existed carry a converted_type instead, such as UTF8 or TIMESTAMP_MILLIS. Writers still set both for older readers. A reader should prefer the logical type when both are present.

A schema element can also carry a field_id: a number that identifies a column across renames. Table formats use it to match columns between files written years apart (ch16). This chapter’s fixture sets none; ch15’s table sets one on every column.

Building it#

The steps above called three parts of the book’s reader: the decoding of each schema element, the rebuild of the tree, and the reading of a value through its logical type. This section builds them, in both languages, and the tabs switch every excerpt on the page between them.

Schema elements#

decode_file_metadata, which ch02 built, gives the footer’s Thrift fields their names. Its field 2 is the schema: a list of structs, each decoded into a SchemaElement by field id:

def schema_element(node: Node) -> SchemaElement:
    s = as_struct(node)
    physical = opt_int(s, 1)
    repetition, converted = opt_int(s, 3), opt_int(s, 6)
    logical_field = s.field(10)
    return SchemaElement(
        name=req_str(s, "SchemaElement", 4, "name", node.span),
        physical_type=PhysicalType(physical) if physical is not None else None,
        type_length=opt_int(s, 2),
        repetition=enum_name("FieldRepetitionType", repetition) if repetition is not None else None,
        num_children=opt_int(s, 5),
        converted_type=enum_name("ConvertedType", converted) if converted is not None else None,
        scale=opt_int(s, 7),
        precision=opt_int(s, 8),
        field_id=opt_int(s, 9),
        logical_type=logical.decode(logical_field.node) if logical_field else None,
        span=node.span,
    )

fn schema_element(node: &Node) -> Result<SchemaElement, MetadataError> {
    let s = as_struct(node)?;
    Ok(SchemaElement {
        name: req_str(s, "SchemaElement", 4, "name", node.span)?,
        physical_type: opt_int(s, 1).map(PhysicalType),
        type_length: opt_int(s, 2),
        repetition: opt_int(s, 3).map(|v| enum_name(EnumName::FieldRepetitionType, v)),
        num_children: opt_int(s, 5),
        converted_type: opt_int(s, 6).map(|v| enum_name(EnumName::ConvertedType, v)),
        scale: opt_int(s, 7),
        precision: opt_int(s, 8),
        field_id: opt_int(s, 9),
        logical_type: s.field(10).and_then(|f| logical::decode(&f.node)),
        span: node.span,
    })
}

Only the name is required. A group has num_children and no physical type; a leaf has a physical type and no children. The logical type is a struct of its own, decoded below.

Rebuilding the tree#

The rebuild is the walk you ran, with checks. It starts at the root and requires the root’s subtree to use up the whole list:

def build(elements: list[SchemaElement]) -> SchemaNode:
    """Rebuild the tree from the footer's flat, depth-first list.

    Read an element; if it says it has ``n`` children, the next ``n`` subtrees are its children,
    each read the same way. The recursion consumes the list from the front, and the whole list
    must be consumed by the root's subtree.
    """
    if not elements:
        raise SchemaError("the schema has no elements, not even a root")
    position = [0]
    root = _subtree(elements, position)
    if position[0] != len(elements):
        raise SchemaError(
            f"the root's subtree ends before the list does; elements from {position[0]} belong to no one"
        )
    return root

pub fn build(elements: &[SchemaElement]) -> Result<SchemaNode, SchemaError> {
    if elements.is_empty() {
        return Err(SchemaError::Empty);
    }
    let mut next = 0;
    let root = subtree(elements, &mut next)?;
    if next != elements.len() {
        return Err(SchemaError::TrailingElements { from: next });
    }
    Ok(root)
}

Each subtree reads an element, and if it has children, reads that many subtrees after it, each the same way. The recursion consumes the list from the front:

def _subtree(elements: list[SchemaElement], position: list[int]) -> SchemaNode:
    index = position[0]
    e = elements[index]
    position[0] += 1
    children = []
    for _ in range(max(e.num_children or 0, 0)):
        if position[0] >= len(elements):
            raise SchemaError(
                f"group {quote(e.name)} (element {index}) claims more children than the schema has elements"
            )
        children.append(_subtree(elements, position))
    if index > 0 and not children and e.physical_type is None:
        raise SchemaError(f"element {index} ({quote(e.name)}) has no children and no physical type")
    return SchemaNode(
        element=index,
        name=e.name,
        repetition=repetition_of(e.repetition),
        physical_type=e.physical_type,
        type_length=e.type_length,
        logical_type=e.logical_type,
        converted_type=e.converted_type,
        field_id=e.field_id,
        span=e.span,
        children=children,
    )

fn subtree(elements: &[SchemaElement], next: &mut usize) -> Result<SchemaNode, SchemaError> {
    let index = *next;
    let e = &elements[index];
    *next += 1;
    let n = e.num_children.unwrap_or(0).max(0) as usize;
    let mut children = Vec::with_capacity(n);
    for _ in 0..n {
        if *next >= elements.len() {
            return Err(SchemaError::TooFewElements {
                group: e.name.clone(),
                element: index,
            });
        }
        children.push(subtree(elements, next)?);
    }
    if index > 0 && n == 0 && e.physical_type.is_none() {
        return Err(SchemaError::NotALeaf {
            name: e.name.clone(),
            element: index,
        });
    }
    Ok(SchemaNode {
        element: index,
        name: e.name.clone(),
        repetition: Repetition::from_name(e.repetition.as_deref()),
        physical_type: e.physical_type,
        type_length: e.type_length,
        logical_type: e.logical_type.clone(),
        converted_type: e.converted_type.clone(),
        field_id: e.field_id,
        span: e.span,
        children,
    })
}

Two checks make a damaged schema an error rather than a wrong tree. A group that claims more children than the list has left is refused. So is a list with elements left over after the root’s subtree ends, which is what the byte you changed in the walk produced.

Maximum levels#

A leaf’s maximum definition level counts the fields on its path that may be absent. Its maximum repetition level counts the fields that repeat:

def max_levels(path: list[str]) -> tuple[int, int]:
    """The largest definition and repetition levels a leaf's values can carry.

    Walk the path from the root's child to the leaf. Every field that may be absent (optional or
    repeated) adds one to the maximum definition level: a value is "defined to level d" when the
    first d such fields are present. Every repeated field adds one to the maximum repetition
    level. Required fields add nothing, because they are always there.
    """
    definition = sum(1 for r in path if r != REQUIRED)
    repetition = sum(1 for r in path if r == REPEATED)
    return definition, repetition

pub fn max_levels(path: &[Repetition]) -> (u32, u32) {
    let mut def = 0;
    let mut rep = 0;
    for r in path {
        match r {
            Repetition::Required => {}
            Repetition::Optional => def += 1,
            Repetition::Repeated => {
                def += 1;
                rep += 1;
            }
        }
    }
    (def, rep)
}

When both are zero, as for every column of tiny.parquet, the column stores no levels at all: its pages hold values and nothing else. ch04 shows what the levels are when they are not zero.

Reading a value through its logical type#

The logical type arrives from the footer as a Thrift union: a struct with exactly one field set, whose id names the type. The logical module decodes it into a value naming the type and its parameters. Applying it is a match on the pair of physical and logical type. Its first cases include the date and the timestamp you read by hand:

def interpret(physical: int, logical: LogicalType, data: bytes) -> str | None:
    """One PLAIN-encoded value, read through its logical type. ``None`` when the bytes do not
    fit, or when the logical type adds nothing to the physical reading.

    A PLAIN statistics value has no length prefix, so a ``BYTE_ARRAY`` value here is the bare
    bytes.
    """
    i32 = struct.unpack("<i", data)[0] if len(data) == 4 else None
    i64 = struct.unpack("<q", data)[0] if len(data) == 8 else None
    name = logical.name
    if physical == 1 and name == "DATE":
        return date_from_days(i32) if i32 is not None else None
    if physical in (1, 2) and name == "INTEGER":
        v = i32 if physical == 1 else i64
        if v is None:
            return None
        return str(v if logical.signed else v % (2**32 if physical == 1 else 2**64))
    if physical == 2 and name == "TIMESTAMP" and logical.unit:
        return timestamp(i64, logical.unit, bool(logical.utc)) if i64 is not None else None

pub fn interpret(physical: PhysicalType, logical: &LogicalType, bytes: &[u8]) -> Option<String> {
    let le_i32 = || bytes.try_into().ok().map(i32::from_le_bytes);
    let le_i64 = || bytes.try_into().ok().map(i64::from_le_bytes);
    match (physical.0, logical) {
        (1, LogicalType::Date) => le_i32().map(|d| date_from_days(i64::from(d))),
        (1, LogicalType::Integer { signed: false, .. }) => le_i32().map(|v| (v as u32).to_string()),
        (2, LogicalType::Integer { signed: false, .. }) => le_i64().map(|v| (v as u64).to_string()),
        (1, LogicalType::Integer { signed: true, .. }) => le_i32().map(|v| v.to_string()),
        (2, LogicalType::Integer { signed: true, .. }) => le_i64().map(|v| v.to_string()),
        (2, LogicalType::Timestamp { utc, unit: Some(u) }) => {
            le_i64().map(|t| timestamp(t, *u, *utc))
        }

The rest follow the same pattern: TIME, then DECIMAL, then text, FLOAT16 and UUID. A decimal can be stored in an INT32, an INT64 or bytes, and only the bytes are big-endian:

    if physical in (1, 2) and name == "DECIMAL":
        v = i32 if physical == 1 else i64
        return decimal(v, logical.scale) if v is not None else None
    if physical in (6, 7) and name == "DECIMAL":
        v = be_twos_complement(data)
        return decimal(v, logical.scale) if v is not None else None

        (1, LogicalType::Decimal { scale, .. }) => le_i32().map(|v| decimal(i128::from(v), *scale)),
        (2, LogicalType::Decimal { scale, .. }) => le_i64().map(|v| decimal(i128::from(v), *scale)),
        (6 | 7, LogicalType::Decimal { scale, .. }) => {
            be_twos_complement(bytes).map(|v| decimal(v, *scale))
        }

The helper for the bytes sign-extends from the top bit of the first byte:

def be_twos_complement(data: bytes) -> int | None:
    """A big-endian two's-complement integer, of any length up to 16 bytes.

    Decimals stored as ``FIXED_LEN_BYTE_ARRAY`` or ``BYTE_ARRAY`` use this. They and ``UUID`` are
    where Parquet puts the most significant byte first; every other number it stores is
    little-endian.
    """
    if not 1 <= len(data) <= 16:
        return None
    # Start from all ones when the top bit says negative, then shift each byte in.
    value = -1 if data[0] & 0x80 else 0
    for b in data:
        value = (value << 8) | b
    return value

pub fn be_twos_complement(bytes: &[u8]) -> Option<i128> {
    if bytes.is_empty() || bytes.len() > 16 {
        return None;
    }
    let negative = bytes[0] & 0x80 != 0;
    let mut v: i128 = if negative { -1 } else { 0 };
    for &b in bytes {
        v = (v << 8) | i128::from(b);
    }
    Some(v)
}

Checking it#

The fixture tests compare the rebuilt schema with pyarrow’s own reading of it: every leaf’s path, physical type and maximum levels. They also apply each column’s logical type to its statistics and compare with the values pyarrow reports:

python3 -m pytest python/tests

cargo test -p parquet-lab --test fixtures

Ask a library#

pyarrow and the parquet crate both rebuild the tree when they open a file, and print it in the Dremel notation. pyarrow’s schema.column(i) and the crate’s SchemaDescriptor give each leaf’s path, physical type and maximum levels, the ones you counted by hand:

# The same schema from pyarrow, which rebuilds the tree and reads the statistics' values for you.
import pyarrow.parquet as pq

md = pq.read_metadata("fixtures/types.parquet")
print(str(md.schema).partition("\n")[2])  # the text form; its first line is the object's repr
for i in range(md.num_columns):
    c, stats = md.schema.column(i), md.row_group(0).column(i).statistics
    levels = f"max levels {c.max_definition_level} {c.max_repetition_level}"
    print(f"{c.path}: {levels}, {c.physical_type}, min {stats.min!r}")

//! The same schema from the `parquet` crate, which rebuilds the tree. Without Arrow, it hands you
//! the statistics as bytes, and reading them through the logical type is yours to do.

use parquet::file::metadata::ParquetMetaDataReader;
use parquet::schema::printer::print_schema;

fn main() {
    let file =
        std::fs::File::open("fixtures/types.parquet").expect("run this from the repository's root");
    let md = ParquetMetaDataReader::new()
        .parse_and_finish(&file)
        .expect("a Parquet file");
    print_schema(&mut std::io::stdout(), md.file_metadata().schema());

    let leaves = md.file_metadata().schema_descr().columns();
    for (c, chunk) in leaves.iter().zip(md.row_group(0).columns()) {
        let levels = format!("max levels {} {}", c.max_def_level(), c.max_rep_level());
        let min = chunk
            .statistics()
            .and_then(|s| s.min_bytes_opt())
            .unwrap_or_default();
        let path = c.path().string();
        println!("{path}: {levels}, {}, min {min:02x?}", c.physical_type());
    }
}

The two part over the statistics. pyarrow reads each minimum through its logical type and hands you a datetime.date and a Decimal. The crate, without its Arrow feature, hands you the bytes the third step read, and the logical reading is yours to do. The two also name the logical types differently: pyarrow’s Int(bitWidth=8, isSigned=true) is the crate’s INTEGER(8,true). pyarrow prints field_id=-1 for a field with no field id, which is every field of this fixture.

In Python, the first run loads pyarrow into the page, a much larger download than the other steps. In Rust the step needs the parquet crate, in walkthroughs/libraries, so it runs in a Codespace or at a desk:

cargo run -q --manifest-path walkthroughs/libraries/Cargo.toml --bin schema_with_a_library

What this cannot tell you#

How nested values are stored. The levels tell a reader how deep a value is defined and where lists begin. This chapter computes their maximums; ch04 decodes them.

Every logical type. The fixture covers the common ones. UUID, FLOAT16, TIME, JSON, ENUM and the newer VARIANT and geospatial types are decoded by name; the reader interprets some of them and reports the rest as unrecognised rather than guessing.

What a legacy writer meant. INT96 timestamps and files with only a converted_type follow conventions that lived outside the specification for years. The reader can decode an INT96 as nanoseconds within a day and a Julian day number. Whether that moment was UTC is something the file does not say.

Whether a writer chose well. Storing a date as a string, or a decimal as a double, is legal and loses information. ch11 looks at those choices from the writer’s side.

Key takeaways#

Problems#

Four, in exercises/python/the_type_system.py, or in Rust in exercises/src/the_type_system.rs. The first three have tests that fail until you solve them. The fourth has no test.

The editor below holds the Python stubs and keeps your edits in this browser. Run the tests runs the graders in the page, the same tests that run at a desk.

3.1 Rebuild the tree. Given each element’s name and number of children, in depth-first order, return every leaf’s path. The test uses the fixtures and hundreds of generated schemas.

3.2 Maximum levels. Given the repetitions along a path, return the maximum definition and repetition levels. The test checks every path up to five fields deep, and the fixtures against pyarrow.

3.3 A decimal from bytes. Read a big-endian two’s-complement integer and place the decimal point. The test compares with pyarrow on the fixture and with the reader on generated values, negative ones included.

3.4 Your own schema. No test: the tables are yours. Print the schema of a Parquet file your systems write (PYTHONPATH=python python3 -m parquet_lab schema FILE, cargo run -p pqlab -- schema FILE, or any Parquet tool). List every column whose logical type is missing where you would expect one: text stored without STRING, dates stored as strings or integers, money stored as DOUBLE, timestamps stored as INT96. For each timestamp, say whether it is UTC or local, and whether that matches what the data means. A good answer names each column, what it should be, and what a reader would get wrong because of the gap.

For the Rust stubs, or to work at a desk, run the chapter’s graders with these commands. Add -k problem_3_1 in Python, or problem_3_1 before the -- in Rust, to run one problem:

python3 -m pytest exercises/python/tests/test_the_type_system.py --problems

cargo test -p exercises --test the_type_system -- --ignored

Where to go next#

• The physical and logical types are specified in the Parquet format repository’s LogicalTypes.md.
• The textual schema notation and the repetition model come from the Dremel paper.
• ch04 decodes the definition and repetition levels whose maximums you computed here.

Built from commit c32f48e.