Parquet, byte by byte · Build a Parquet reader while you read about one

ch13Modular encryption

What you build: A reader that recognises encrypted modules and reports what stays visible.

The question#

What can a reader still see in an encrypted file, and what changes for it?

Everything so far assumed a reader could read every byte. Parquet can encrypt a file, and does so in pieces: the footer, and for each protected column every page header and every page, each encrypted on its own. A reader with the right keys decrypts what it needs. A reader without them sees a file whose shape is still visible and whose protected contents are not. This chapter reads two encrypted files without their keys, and reports exactly where the reader stops.

The experiment#

plaintext-footer.parquet and encrypted-footer.parquet hold the same twelve orders, with the same keys. email is encrypted with one key and amount_cents with another; order_id and country are not encrypted. The files differ in one choice: whether the footer is encrypted too.

Each step below is a few lines of code. In Python, run them in the page, change them with Edit, and run them again. In Rust, open them in a Codespace, or run one at a desk with cargo run -p walkthroughs --bin and its name.

Footer modes. A Parquet file starts and ends with its magic, as ch02 found. Read the first and last four bytes of both files:

# The first and last four bytes of two encrypted files: the magic names the footer mode.
for name in ["plaintext-footer", "encrypted-footer"]:  # try adding "tiny"
    data = open(f"fixtures/{name}.parquet", "rb").read()
    print(f"{name}.parquet starts {data[:4]} and ends {data[-4:]}")
//! The first and last four bytes of two encrypted files: the magic names the footer mode.

fn main() {
    // Try adding "tiny".
    for name in ["plaintext-footer", "encrypted-footer"] {
        let path = format!("fixtures/{name}.parquet");
        let data = std::fs::read(path).expect("run this from the repository's root");
        let text = |b: &[u8]| String::from_utf8_lossy(b).into_owned();
        let (first, last) = (text(&data[..4]), text(&data[data.len() - 4..]));
        println!("{name}.parquet starts {first:?} and ends {last:?}");
    }
}

The plaintext-footer file starts and ends with PAR1, like every file so far, and a reader that knows nothing of encryption opens it as usual. The encrypted-footer file has PARE at both ends, and such a reader refuses it at the magic. Add "tiny" to the list of files: it ends with PAR1 too, so the magic alone cannot tell a plaintext footer from a file with no encryption at all. Problem 13.2 asks you to tell the three apart.

Keys by name#

Neither file holds a key. Each encrypted piece carries key metadata, which names a key without revealing it. pyarrow writes it as JSON, so a search of the raw bytes finds it.

Keys by name. Find every piece of key metadata in both files, and print the key it names:

# Key metadata names a key without holding it: pyarrow writes it as JSON.
import json, re
for name in ["plaintext-footer", "encrypted-footer"]:
    data = open(f"fixtures/{name}.parquet", "rb").read()
    found = re.findall(rb'\{"keyMaterialType".*?\}', data)
    print(f"{name}.parquet, keys named in the clear: {len(found)}")
    for text in found:
        key = json.loads(text)  # try printing all of it
        kind = "footer" if key["isFooterKey"] else "column"
        master, wrapped = key["masterKeyID"], key["wrappedDEK"]
        print(f"  {kind} key {master!r}, wrapped data key {wrapped}")
//! Key metadata names a key without holding it: pyarrow writes it as JSON.

/// A string field of flat JSON: what follows `"name":"`, up to the next quote.
fn field<'a>(json: &'a str, name: &str) -> &'a str {
    let start = json.find(&format!("\"{name}\":\"")).expect("the field") + name.len() + 4;
    &json[start..start + json[start..].find('"').unwrap()]
}

fn main() {
    for name in ["plaintext-footer", "encrypted-footer"] {
        let path = format!("fixtures/{name}.parquet");
        let data = std::fs::read(path).expect("run this from the repository's root");
        let text = String::from_utf8_lossy(&data); // the ciphertext comes out as nonsense
        let found: Vec<&str> = text
            .match_indices("{\"keyMaterialType\"")
            .map(|(at, _)| &text[at..at + text[at..].find('}').unwrap() + 1])
            .collect();
        println!("{name}.parquet, keys named in the clear: {}", found.len());
        for json in found {
            // Try printing all of it.
            let footer = json.contains("\"isFooterKey\":true");
            let kind = if footer { "footer" } else { "column" };
            let master = field(json, "masterKeyID");
            let wrapped = field(json, "wrappedDEK");
            println!("  {kind} key {master:?}, wrapped data key {wrapped}");
        }
    }
}

The plaintext-footer file names three keys in the clear: one for each encrypted column, and one for the footer. The encrypted-footer file names only the footer key. The columns’ key metadata is still there, inside the footer, and the footer is encrypted. Print all of the JSON: it also records whether the data key was wrapped once or twice and, for the footer key, which key service to ask.

The scheme is envelope encryption. Each column is encrypted with a random data key. That data key is wrapped, encrypted, with a master key that stays inside a key management service: the wrappedDEK the step printed is the wrapped data key. A reader sends it to the service, which unwraps it only for a reader allowed to have it. Different columns can use different master keys, so one reader may see amount_cents and not email.

The fixtures use a toy key service that wraps by XOR with master keys published in fixtures/generate.py. It protects nothing, on purpose: the files exist to show what encryption hides from a reader without keys, and anyone may regenerate them.

Modules#

Parquet encrypts modules, not files. Every encrypted piece is laid out the same way:

[length: u32, little-endian]   the bytes that follow
[nonce: 12 bytes]              never reused under one key
[ciphertext]                   the module, encrypted with AES
[tag: 16 bytes]                AES-GCM's check that nothing was changed

One module. The plaintext footer still says where email’s column chunk is. Read the first module of the chunk from its length prefix:

# One encrypted module, read from its length prefix: the first of email's column chunk.
import sys
sys.path.insert(0, "python")  # the book's reader, from the repository's python/ directory
from parquet_lab import report

data = open("fixtures/plaintext-footer.parquet", "rb").read()
md = report.open_bytes(data)  # a plaintext footer: found and decoded as in ch02
chunk = md.row_groups[0].columns[2]  # email; try 3, amount_cents
path, start, size = chunk.dotted_path(), chunk.data_page_offset, chunk.total_compressed_size
print(f"{path}: bytes {start} to {start + size}, statistics {chunk.statistics}")
at = start  # try the byte where the next module starts
n = int.from_bytes(data[at:at + 4], "little")  # the length: the bytes that follow it
print(f"a module at byte {at}: length {n}")
print("  nonce:", data[at + 4:at + 16].hex(" "))
print(f"  ciphertext: {n - 12 - 16} bytes")
print("  tag:", data[at + 4 + n - 16:at + 4 + n].hex(" "))
print(f"the next module starts at byte {at + 4 + n}")
//! One encrypted module, read from its length prefix: the first of email's column chunk.

use parquet_lab::report::open_bytes;

/// Bytes as two hex digits each, spaced, as Python's `bytes.hex(" ")` writes them.
fn hex(bytes: &[u8]) -> String {
    let each: Vec<String> = bytes.iter().map(|b| format!("{b:02x}")).collect();
    each.join(" ")
}

fn main() {
    let path = "fixtures/plaintext-footer.parquet";
    let data = std::fs::read(path).expect("run this from the repository's root");
    let md = open_bytes(&data).expect("a footer"); // decoded as in ch02
    let chunk = &md.row_groups[0].columns[2]; // email; try 3, amount_cents
    let (start, size) = (chunk.data_page_offset, chunk.total_compressed_size);
    let stats = &chunk.statistics;
    println!(
        "email: bytes {start} to {}, statistics {stats:?}",
        start + size
    );
    let at = start as usize; // try the byte where the next module starts
                             // The length: the bytes that follow it.
    let n = u32::from_le_bytes(data[at..at + 4].try_into().unwrap()) as usize;
    println!("a module at byte {at}: length {n}");
    println!("  nonce: {}", hex(&data[at + 4..at + 16]));
    println!("  ciphertext: {} bytes", n - 12 - 16);
    println!("  tag: {}", hex(&data[at + 4 + n - 16..at + 4 + n]));
    println!("the next module starts at byte {}", at + 4 + n);
}

The footer gives the chunk’s place and size, but no statistics: those are in the encrypted copy of the column’s metadata. The module’s length counts the nonce, the ciphertext and the tag, and the next module starts where it ends. Set at to that byte and run the step again: the second module ends exactly where the chunk ends. The first module is the page’s header and the second the page. Change the column to amount_cents and read its first module the same way. Walking every module of a chunk is problem 13.1.

Encrypting each page separately keeps what earlier chapters built. A reader with the keys can still fetch one page from the middle of a file and decrypt it alone, as ch10 fetched pages. And a reader without the keys can still find every module, by its length prefix, the way ch06 walked page headers.

The algorithm is AES-GCM. It encrypts, and its tag also authenticates: a module changed by a single bit fails to decrypt instead of decrypting to something wrong. Each module is also bound to its place in the file, its row group, column and page number, through the data GCM authenticates alongside the ciphertext, so a module cannot be moved from one place to another unnoticed. A second algorithm, AES_GCM_CTR_V1, authenticates the metadata but encrypts pages with plain counter mode, which is faster and leaves the pages unauthenticated.

The file, module by module#

The panel lays out both files as the reader parsed them, in the structure and byte views of ch02’s panel, and lists the columns as a reader without keys finds them. Select a column’s name to mark its bytes.

This experiment runs the book's Parquet reader in your browser, compiled to WebAssembly. It needs JavaScript. The same reader runs at a desk: see Appendix A.

Try these:

  1. Select email. Its chunk is the two modules the step read, a page header and a page. The structure view splits each into its length, nonce, ciphertext and tag.
  2. Find the signature. In the structure view, after the FileMetaData, a nonce and a tag sign the footer.
  3. Switch to encrypted-footer.parquet. No column can be located. The structure view shows a plaintext FileCryptoMetaData and one encrypted module where the FileMetaData should be.
A reader without keysWhatDetail
can seeThe encryption algorithmAES_GCM_V1
can seeThe footer’s signature, which only the footer key can check28 bytes
can seeThe schemaorder_id, country, email, amount_cents
can seeThe number of rows12
can seeThe values of order_id1, 2, 3, 4, …
can seeThe statistics of order_id1 to 12
can seeThe values of country“UK”, “SE”, “UK”, “PL”, …
can seeThe statistics of country“PL” to “US”
can seeWhich key protects emailpii
can seeWhich key protects amount_centsfinance
cannot seeThe values of emailcolumn email is encrypted with its own key, and this reader has no keys
cannot seeThe statistics of emailwithheld from the plaintext footer
cannot seeThe values of amount_centscolumn amount_cents is encrypted with its own key, and this reader has no keys
cannot seeThe statistics of amount_centswithheld from the plaintext footer

Computed by the reader from fixtures/plaintext-footer.parquet (2192 bytes).

With a plaintext footer, the file still starts and ends with PAR1, and a reader that knows nothing of encryption reads the unencrypted columns as usual. The encrypted columns keep a reduced ColumnMetaData in the footer: their type, encodings, codec, sizes and offsets, but no statistics. The full metadata, statistics included, is itself an encrypted module inside the footer. Readers without keys therefore cannot skip by those columns’ statistics, and cannot read them at all.

The footer ends with a signature: a nonce and a GCM tag computed with the footer key. A reader with that key can check that nobody changed the footer. A reader without it cannot, and has to trust the footer as it is.

A reader without keysWhatDetail
can seeThat it is Parquet, with an encrypted footerPARE
can seeThe encryption algorithmAES_GCM_V1
can seeThe footer key’s namefooter
can seeThe encrypted footer’s size954 bytes
cannot seeThe schemainside the encrypted footer
cannot seeThe number of rowsinside the encrypted footer
cannot seeWhere any column isinside the encrypted footer
cannot seeAny statisticsinside the encrypted footer
cannot seeAny values, even of columns not encryptedinside the encrypted footer

Computed by the reader from fixtures/encrypted-footer.parquet (2123 bytes).

With an encrypted footer, the file starts and ends with PARE, and the footer is a small plaintext FileCryptoMetaData, naming the algorithm and the footer key, followed by the real FileMetaData as one module. Even the unencrypted columns cannot be read: their bytes are there in plain form, but the footer that says where they are is not.

What still leaks#

Encryption hides values, not shape. From the steps, the tables above and the panel, a reader without keys still learns:

  • the file’s size, and in a plaintext footer every column chunk’s size and offset;
  • how many pages each encrypted column has, and how long each is. AES-GCM does not change a module’s length, so the ciphertext is exactly as long as the page, and page lengths can say something about the values;
  • which columns are protected, and by which named keys.

The encrypted footer hides more of the shape, at the price of making the whole file unreadable without the footer key.

Building it#

The steps read the magic, the key metadata and one module by hand, and stopped where a key would be needed. The reader does the same for every module, in both footer modes, and says where it stops.

Walking modules#

A module is read from its length prefix alone, as the step read one, and an encrypted column chunk is a run of them:

From the implementationpython/parquet_lab/crypto.py
def read_module(r: ByteReader) -> Module:
    """Read the module that starts at ``r``, from its length prefix alone."""
    start = r.offset()
    n = r.read_le_u32()
    if n < NONCE_LEN + TAG_LEN:
        why = f"a module at offset {start} claims {n} bytes, fewer than a nonce and a tag"
        raise CryptoError(why)
    _, body = r.read_bytes(n)
    return Module(
        span=Span(start, body.end),
        length=Span(start, start + 4),
        nonce=Span(body.start, body.start + NONCE_LEN),
        ciphertext=Span(body.start + NONCE_LEN, body.end - TAG_LEN),
        tag=Span(body.end - TAG_LEN, body.end),
    )

From the implementationcrates/parquet-lab/src/crypto.rs
pub fn read_module(r: &mut ByteReader) -> Result<Module, String> {
    let start = r.offset();
    let len = r.read_le_u32().map_err(|e| e.to_string())? as u64;
    if len < NONCE_LEN + TAG_LEN {
        return Err(format!(
            "a module at offset {start} claims {len} bytes, fewer than a nonce and a tag"
        ));
    }
    let (_, body) = r.read_bytes(len as usize).map_err(|e| e.to_string())?;
    Ok(Module {
        span: Span::new(start, body.end),
        length: Span::new(start, start + 4),
        nonce: Span::new(body.start, body.start + NONCE_LEN),
        ciphertext: Span::new(body.start + NONCE_LEN, body.end - TAG_LEN),
        tag: Span::new(body.end - TAG_LEN, body.end),
    })
}

A file that ends with PARE has a footer in two parts. The reader decodes the first, the plaintext FileCryptoMetaData, for the algorithm and the footer key’s metadata, and reads the rest as one module, which must end where the footer does:

From the implementationpython/parquet_lab/crypto.py
def encrypted_footer(file: bytes) -> EncryptedFooter:
    """Read an encrypted-footer file's footer, as far as a reader without its key can."""
    size = len(file)
    if size < 12 or file[:4] != MAGIC_ENCRYPTED or file[-4:] != MAGIC_ENCRYPTED:
        why = "not an encrypted-footer file: it does not start and end with PARE"
        raise CryptoError(why)
    length = int.from_bytes(file[-8:-4], "little")
    footer = footer_span(size, length)
    r = ByteReader(file[footer.start : footer.end], footer.start)
    try:
        crypto_metadata = read_struct(r)
    except ValueError as e:
        raise CryptoError(f"FileCryptoMetaData: {e}") from e
    s = as_struct(crypto_metadata)
    algorithm = union_member(s.field(1).node) if s.field(1) else "UNKNOWN"
    key = s.field(2)
    key_metadata = key.node.value if key and isinstance(key.node.value, bytes) else None
    module = read_module(r)
    if not r.is_at_end():
        left = footer.end - r.offset()
        why = f"{left} bytes follow the encrypted FileMetaData inside the footer"
        raise CryptoError(why)
    return EncryptedFooter(footer, crypto_metadata, algorithm, key_metadata, module)
From the implementationcrates/parquet-lab/src/crypto.rs
pub fn encrypted_footer(file: &[u8]) -> Result<EncryptedFooter, String> {
    let n = file.len();
    if n < 12 || file[..4] != MAGIC_ENCRYPTED || file[n - 4..] != MAGIC_ENCRYPTED {
        let why = "not an encrypted-footer file: it does not start and end with PARE";
        return Err(why.into());
    }
    let length = u32::from_le_bytes(file[n - 8..n - 4].try_into().unwrap());
    let footer = crate::format::footer_span(n as u64, length).map_err(|e| e.to_string())?;
    let bytes = &file[footer.start as usize..footer.end as usize];
    let mut r = ByteReader::new(bytes, footer.start);
    let crypto_metadata = read_struct(&mut r).map_err(|e| format!("FileCryptoMetaData: {e}"))?;
    let s = crate::metadata::as_struct(&crypto_metadata).map_err(|e| e.to_string())?;
    let algorithm = s
        .field(1)
        .map(|f| crate::metadata::union_member(&f.node))
        .unwrap_or_else(|| "UNKNOWN".into());
    let key_metadata = match s.field(2).map(|f| &f.node.value) {
        Some(crate::thrift::Value::Binary(b)) => Some(b.clone()),
        _ => None,
    };
    let module = read_module(&mut r)?;
    if !r.is_at_end() {
        return Err(format!(
            "{} bytes follow the encrypted FileMetaData inside the footer",
            footer.end - r.offset()
        ));
    }
    Ok(EncryptedFooter {
        footer,
        crypto_metadata,
        algorithm,
        key_metadata,
        module,
    })
}

In the step that read one module, open_bytes decoded the plaintext footer with the reader’s footer decoder, the metadata module. That decoder refuses bytes left over after the FileMetaData: they mean the trailer’s length and the structure disagree. An encrypted file with a plaintext footer leaves exactly a signature’s worth, and says it uses encryption:

From the implementationpython/parquet_lab/metadata.py
    # An encrypted file with a plaintext footer (ch13) signs it: the FileMetaData is followed by
    # a 12-byte nonce and a 16-byte AES-GCM tag, and the footer length counts them.
    footer_signature = None
    if not r.is_at_end():
        rest = Span(r.offset(), base + len(footer))
        if s.field(8) is None or rest.length != FOOTER_SIGNATURE_LEN:
            raise TrailingBytes(rest)
        footer_signature = rest
From the implementationcrates/parquet-lab/src/metadata.rs
    // An encrypted file with a plaintext footer (ch13) signs it: the FileMetaData is followed by
    // a 12-byte nonce and a 16-byte AES-GCM tag, and the footer length counts them.
    let mut footer_signature = None;
    if !r.is_at_end() {
        let rest = Span::new(r.offset(), base + footer.len() as u64);
        if s.field(8).is_some() && rest.len() == FOOTER_SIGNATURE_LEN {
            footer_signature = Some(rest);
        } else {
            return Err(MetadataError::TrailingBytes { span: rest });
        }
    }

Refusing what it cannot read#

The column reader refuses an encrypted column by name, rather than decoding ciphertext as if it were a page, which is what a library without encryption does under Ask a library below:

From the implementationpython/parquet_lab/column.py
def refuse_encrypted(chunk: ColumnChunk) -> None:
    """An encrypted column chunk's pages are encrypted modules (ch13). Without its key
    they cannot be read, and the reader says so rather than decoding ciphertext."""
    if chunk.crypto is not None:
        key = "its own key" if chunk.crypto.with_column_key else "the footer key"
        path = chunk.dotted_path()
        why = f"column {path} is encrypted with {key}, and this reader has no keys"
        raise ColumnError(why)

From the implementationcrates/parquet-lab/src/column.rs
fn refuse_encrypted(chunk: &ColumnChunk) -> Result<(), ColumnError> {
    match &chunk.crypto {
        Some(c) => err(format!(
            "column {} is encrypted with {}, and this reader has no keys",
            chunk.dotted_path(),
            if c.with_column_key {
                "its own key"
            } else {
                "the footer key"
            }
        )),
        None => Ok(()),
    }
}

Checking it#

The fixtures were written by pyarrow with the keys, and the manifests describe them as pyarrow saw them with the keys. The tests hold the keyless reader to that description: the encrypted footer module is exactly the size pyarrow reports for the footer, the column chunks pyarrow found fill exactly the bytes before it, the unencrypted columns decode to pyarrow’s rows, and the encrypted columns’ modules tile their chunks.

python3 -m pytest python/tests -k encrypt
cargo test -p parquet-lab --test fixtures encrypt

Encryption uses a fresh nonce for every module and a fresh data key for every file, so these two fixtures cannot be regenerated byte for byte. They were written once; the generator’s check decrypts them with the published keys and confirms they still hold exactly the intended rows.

Ask a library#

At work a library reads these files, with keys from a key service. Without keys, pyarrow and the parquet crate stop where the book’s reader stops, though not in the same way:

# pyarrow without keys: a plaintext footer opens, an encrypted one does not.
import pyarrow.parquet as pq

path = "fixtures/plaintext-footer.parquet"
metadata = pq.read_metadata(path)
print(f"plaintext-footer.parquet: {metadata.num_rows} rows")
for i in [0, 1]:  # never 2 or 3: pyarrow stops Python on an encrypted column's metadata
    chunk = metadata.row_group(0).column(i)
    s = chunk.statistics
    print(f"  {chunk.path_in_schema}: statistics {s.min} to {s.max}")
for columns in [["order_id", "country"], ["email"]]:  # try ["amount_cents"]
    try:
        table = pq.read_table(path, columns=columns)
        print(f"  read {columns}: {table.num_rows} rows")
    except OSError as e:
        print(f"  read {columns}: {e}")
try:
    pq.read_metadata("fixtures/encrypted-footer.parquet")
except OSError as e:
    print(f"encrypted-footer.parquet: {e}")
//! The `parquet` crate without keys: a plaintext footer opens, an encrypted one does not.

use std::fs::File;

use parquet::file::reader::{FileReader, SerializedFileReader};
use parquet::file::statistics::Statistics;
use parquet::schema::types::Type;

/// A column's bounds, if the footer holds them where this reader can read them.
fn bounds(statistics: Option<&Statistics>) -> String {
    match statistics {
        Some(Statistics::Int64(s)) => {
            format!("{} to {}", s.min_opt().unwrap(), s.max_opt().unwrap())
        }
        Some(Statistics::ByteArray(s)) => {
            let (min, max) = (s.min_opt().unwrap(), s.max_opt().unwrap());
            format!(
                "{:?} to {:?}",
                min.as_utf8().unwrap(),
                max.as_utf8().unwrap()
            )
        }
        _ => "none".to_string(),
    }
}

fn main() {
    let path = "fixtures/plaintext-footer.parquet";
    let file = File::open(path).expect("run this from the repository's root");
    let reader = SerializedFileReader::new(file).expect("a plaintext footer");
    let metadata = reader.metadata();
    let rows = metadata.file_metadata().num_rows();
    println!("plaintext-footer.parquet: {rows} rows");
    for chunk in metadata.row_group(0).columns() {
        let name = chunk.column_path().string();
        println!("  {name}: statistics {}", bounds(chunk.statistics()));
    }
    // Without its encryption feature the crate does not know email's pages are encrypted.
    let fields = metadata.file_metadata().schema().get_fields();
    let column = &fields[2]; // email; try 3, amount_cents
    let projection = Type::group_type_builder("schema").with_fields(vec![column.clone()]);
    let mut rows = reader
        .get_row_iter(Some(projection.build().unwrap()))
        .expect("rows");
    match rows.next().expect("a row") {
        Ok(row) => println!("  {}'s first row: {row}", column.name()),
        Err(e) => println!("  {}'s first row: {e}", column.name()),
    }

    let file = File::open("fixtures/encrypted-footer.parquet").unwrap();
    match SerializedFileReader::new(file) {
        Ok(_) => println!("encrypted-footer.parquet: opened"),
        Err(e) => println!("encrypted-footer.parquet: {e}"),
    }
}

Both open the plaintext footer and count the rows. pyarrow gives the statistics of order_id and country and reads those two columns. Asked for email, it refuses, because it cannot decrypt the column’s metadata. The crate, built here without its encryption feature, finds no statistics for the two encrypted columns, as the book’s reader found none. But it does not know that email’s pages are encrypted: it reads the first module’s length prefix as the start of a page header, and fails on a field the header lacks. The book’s reader refuses the column by name before it reads a page. Neither library opens the encrypted footer without keys.

The Python step keeps away from one call. pyarrow’s metadata for an encrypted column, metadata.row_group(0).column(2), needs that column’s key. Without the key, pyarrow does not raise an error: it stops the whole Python process, and in the page that stops every step. A reader with keys passes decryption properties from pyarrow.parquet.encryption, as the pyarrow documentation under Where to go next shows. Ask for the other encrypted column, amount_cents, instead of email: both libraries fail on it as they did on email.

In Python, the first run loads pyarrow into the page, a much larger download than the other steps. In Rust the step needs the parquet crate, in walkthroughs/libraries, so it runs in a Codespace or at a desk:

cargo run -q --manifest-path walkthroughs/libraries/Cargo.toml --bin encryption_with_a_library

What this cannot tell you#

How to decrypt. This reader stops where the keys would be needed. Decrypting needs AES, GCM, the service that unwraps data keys, and the exact authenticated data for each module; the format specifies all of them, and this book does not implement them.

Whether to encrypt. Encryption protects files wherever they are copied. Access control on the storage protects them only where they are stored. Which risk matters is a question about the data, not the format.

How table formats use it. Engines that manage many files, ch14’s subject, decide which columns to encrypt and hold the keys; the files only carry the result.

Key takeaways#

  • Parquet encrypts modules, not files. The footer and every page header and page are encrypted separately, so pages can still be read one at a time.
  • A plaintext footer keeps the file readable without keys, except for the encrypted columns, whose values and statistics are hidden.
  • An encrypted footer hides the whole layout. Without the footer key even unencrypted columns cannot be found.
  • Files name keys; they never hold them. Data keys are wrapped by master keys that stay in a key service.
  • Encryption hides values, not shape. Sizes, page counts and page lengths remain visible.

Problems#

Three, in exercises/python/modular_encryption.py, or in Rust in exercises/src/modular_encryption.rs. The first two have tests. The third has none.

The editor below holds the Python stubs and keeps your edits in this browser. Run the tests runs the graders in the page, the same tests that run at a desk.

Here you can edit this chapter's problems and run their tests in your browser, in Python. It needs JavaScript. The same problems run at a desk: see Appendix A.

13.1 Find the modules. Split an encrypted column chunk into its modules by their length prefixes. The test compares your modules with the reader’s for both encrypted columns, and with generated chunks.

13.2 Which footer mode? Tell an unencrypted file, a plaintext footer and an encrypted footer apart. The test checks every fixture.

13.3 Your own threat model. No test: the data is yours. Pick a table with sensitive columns and decide which to encrypt, and whether the footer should be. List what a reader without keys would still learn in your choice, from the sizes, offsets and page counts this chapter showed. A good answer names one thing the plaintext footer reveals that matters for your data, and says whether that justifies an encrypted footer and the cost of making every reader hold a key.

For the Rust stubs, or to work at a desk, run the chapter’s graders with these commands. Add -k problem_13_1 in Python, or problem_13_1 before the -- in Rust, to run one problem:

python3 -m pytest exercises/python/tests/test_modular_encryption.py --problems
cargo test -p exercises --test modular_encryption -- --ignored

Where to go next#

  • Modular encryption is specified in the format repository’s Encryption.md.
  • pyarrow’s encryption documentation shows how a real key service is plugged in.
  • AES-GCM is specified in NIST Special Publication 800-38D.
  • ch14 turns from one file to the many that make a table.
By Chris Snow, in collaboration with Claude (Anthropic) · Prose and figures: CC BY-NC 4.0 · Code: Apache 2.0

Built from commit c32f48e.