ch13Modular encryption
What you build: A reader that recognises encrypted modules and reports what stays visible.
The question#
What can a reader still see in an encrypted file, and what changes for it?
Everything so far assumed a reader could read every byte. Parquet can encrypt a file, and does so in pieces: the footer, and for each protected column every page header and every page, each encrypted on its own. A reader with the right keys decrypts what it needs. A reader without them sees a file whose shape is still visible and whose protected contents are not. This chapter reads two encrypted files without their keys, and reports exactly where the reader stops.
The experiment#
Two files, two footer modes#
plaintext-footer.parquet and encrypted-footer.parquet hold the same twelve orders, with the
same keys. email is encrypted with one key and amount_cents with another; order_id and
country are not encrypted. The files differ in one choice: whether the footer is encrypted too.
Each step below is a few lines of code. In Python, run them in the page, change them with
Edit, and run them again. In Rust, open them in a Codespace, or run one at a desk with
cargo run -p walkthroughs --bin and its name.
Footer modes. A Parquet file starts and ends with its magic, as ch02 found. Read the first and last four bytes of both files:
# The first and last four bytes of two encrypted files: the magic names the footer mode.
for name in ["plaintext-footer", "encrypted-footer"]: # try adding "tiny"
data = open(f"fixtures/{name}.parquet", "rb").read()
print(f"{name}.parquet starts {data[:4]} and ends {data[-4:]}")
//! The first and last four bytes of two encrypted files: the magic names the footer mode.
fn main() {
// Try adding "tiny".
for name in ["plaintext-footer", "encrypted-footer"] {
let path = format!("fixtures/{name}.parquet");
let data = std::fs::read(path).expect("run this from the repository's root");
let text = |b: &[u8]| String::from_utf8_lossy(b).into_owned();
let (first, last) = (text(&data[..4]), text(&data[data.len() - 4..]));
println!("{name}.parquet starts {first:?} and ends {last:?}");
}
}
The plaintext-footer file starts and ends with PAR1, like every file so far, and a reader that
knows nothing of encryption opens it as usual. The encrypted-footer file has PARE at both ends,
and such a reader refuses it at the magic. Add "tiny" to the list of files: it ends with
PAR1 too, so the magic alone cannot tell a plaintext footer from a file with no encryption at
all. Problem 13.2 asks you to tell the three apart.
Keys by name#
Neither file holds a key. Each encrypted piece carries key metadata, which names a key without revealing it. pyarrow writes it as JSON, so a search of the raw bytes finds it.
Keys by name. Find every piece of key metadata in both files, and print the key it names:
# Key metadata names a key without holding it: pyarrow writes it as JSON.
import json, re
for name in ["plaintext-footer", "encrypted-footer"]:
data = open(f"fixtures/{name}.parquet", "rb").read()
found = re.findall(rb'\{"keyMaterialType".*?\}', data)
print(f"{name}.parquet, keys named in the clear: {len(found)}")
for text in found:
key = json.loads(text) # try printing all of it
kind = "footer" if key["isFooterKey"] else "column"
master, wrapped = key["masterKeyID"], key["wrappedDEK"]
print(f" {kind} key {master!r}, wrapped data key {wrapped}")
//! Key metadata names a key without holding it: pyarrow writes it as JSON.
/// A string field of flat JSON: what follows `"name":"`, up to the next quote.
fn field<'a>(json: &'a str, name: &str) -> &'a str {
let start = json.find(&format!("\"{name}\":\"")).expect("the field") + name.len() + 4;
&json[start..start + json[start..].find('"').unwrap()]
}
fn main() {
for name in ["plaintext-footer", "encrypted-footer"] {
let path = format!("fixtures/{name}.parquet");
let data = std::fs::read(path).expect("run this from the repository's root");
let text = String::from_utf8_lossy(&data); // the ciphertext comes out as nonsense
let found: Vec<&str> = text
.match_indices("{\"keyMaterialType\"")
.map(|(at, _)| &text[at..at + text[at..].find('}').unwrap() + 1])
.collect();
println!("{name}.parquet, keys named in the clear: {}", found.len());
for json in found {
// Try printing all of it.
let footer = json.contains("\"isFooterKey\":true");
let kind = if footer { "footer" } else { "column" };
let master = field(json, "masterKeyID");
let wrapped = field(json, "wrappedDEK");
println!(" {kind} key {master:?}, wrapped data key {wrapped}");
}
}
}
The plaintext-footer file names three keys in the clear: one for each encrypted column, and one for the footer. The encrypted-footer file names only the footer key. The columns’ key metadata is still there, inside the footer, and the footer is encrypted. Print all of the JSON: it also records whether the data key was wrapped once or twice and, for the footer key, which key service to ask.
The scheme is envelope encryption. Each column is encrypted with a random data key. That data key
is wrapped, encrypted, with a master key that stays inside a key management service: the
wrappedDEK the step printed is the wrapped data key. A reader sends it to the service, which
unwraps it only for a reader allowed to have it. Different columns can use different master
keys, so one reader may see amount_cents and not email.
The fixtures use a toy key service that wraps by XOR with master keys published in
fixtures/generate.py. It protects nothing, on purpose: the files exist to show what encryption
hides from a reader without keys, and anyone may regenerate them.
Modules#
Parquet encrypts modules, not files. Every encrypted piece is laid out the same way:
[length: u32, little-endian] the bytes that follow
[nonce: 12 bytes] never reused under one key
[ciphertext] the module, encrypted with AES
[tag: 16 bytes] AES-GCM's check that nothing was changedOne module. The plaintext footer still says where email’s column chunk is. Read the first
module of the chunk from its length prefix:
# One encrypted module, read from its length prefix: the first of email's column chunk.
import sys
sys.path.insert(0, "python") # the book's reader, from the repository's python/ directory
from parquet_lab import report
data = open("fixtures/plaintext-footer.parquet", "rb").read()
md = report.open_bytes(data) # a plaintext footer: found and decoded as in ch02
chunk = md.row_groups[0].columns[2] # email; try 3, amount_cents
path, start, size = chunk.dotted_path(), chunk.data_page_offset, chunk.total_compressed_size
print(f"{path}: bytes {start} to {start + size}, statistics {chunk.statistics}")
at = start # try the byte where the next module starts
n = int.from_bytes(data[at:at + 4], "little") # the length: the bytes that follow it
print(f"a module at byte {at}: length {n}")
print(" nonce:", data[at + 4:at + 16].hex(" "))
print(f" ciphertext: {n - 12 - 16} bytes")
print(" tag:", data[at + 4 + n - 16:at + 4 + n].hex(" "))
print(f"the next module starts at byte {at + 4 + n}")
//! One encrypted module, read from its length prefix: the first of email's column chunk.
use parquet_lab::report::open_bytes;
/// Bytes as two hex digits each, spaced, as Python's `bytes.hex(" ")` writes them.
fn hex(bytes: &[u8]) -> String {
let each: Vec<String> = bytes.iter().map(|b| format!("{b:02x}")).collect();
each.join(" ")
}
fn main() {
let path = "fixtures/plaintext-footer.parquet";
let data = std::fs::read(path).expect("run this from the repository's root");
let md = open_bytes(&data).expect("a footer"); // decoded as in ch02
let chunk = &md.row_groups[0].columns[2]; // email; try 3, amount_cents
let (start, size) = (chunk.data_page_offset, chunk.total_compressed_size);
let stats = &chunk.statistics;
println!(
"email: bytes {start} to {}, statistics {stats:?}",
start + size
);
let at = start as usize; // try the byte where the next module starts
// The length: the bytes that follow it.
let n = u32::from_le_bytes(data[at..at + 4].try_into().unwrap()) as usize;
println!("a module at byte {at}: length {n}");
println!(" nonce: {}", hex(&data[at + 4..at + 16]));
println!(" ciphertext: {} bytes", n - 12 - 16);
println!(" tag: {}", hex(&data[at + 4 + n - 16..at + 4 + n]));
println!("the next module starts at byte {}", at + 4 + n);
}
The footer gives the chunk’s place and size, but no statistics: those are in the encrypted copy
of the column’s metadata. The module’s length counts the nonce, the ciphertext and the tag, and
the next module starts where it ends. Set at to that byte and run the step again: the second
module ends exactly where the chunk ends. The first module is the page’s header and the second
the page. Change the column to amount_cents and read its first module the same way. Walking
every module of a chunk is problem 13.1.
Encrypting each page separately keeps what earlier chapters built. A reader with the keys can still fetch one page from the middle of a file and decrypt it alone, as ch10 fetched pages. And a reader without the keys can still find every module, by its length prefix, the way ch06 walked page headers.
The algorithm is AES-GCM. It encrypts, and its tag also authenticates: a module changed by a
single bit fails to decrypt instead of decrypting to something wrong. Each module is also bound
to its place in the file, its row group, column and page number, through the data GCM
authenticates alongside the ciphertext, so a module cannot be moved from one place to another
unnoticed. A second algorithm, AES_GCM_CTR_V1, authenticates the metadata but encrypts pages
with plain counter mode, which is faster and leaves the pages unauthenticated.
The file, module by module#
The panel lays out both files as the reader parsed them, in the structure and byte views of ch02’s panel, and lists the columns as a reader without keys finds them. Select a column’s name to mark its bytes.
This experiment runs the book's Parquet reader in your browser, compiled to WebAssembly. It needs JavaScript. The same reader runs at a desk: see Appendix A.
Try these:
- Select
email. Its chunk is the two modules the step read, a page header and a page. The structure view splits each into its length, nonce, ciphertext and tag. - Find the signature. In the structure view, after the
FileMetaData, a nonce and a tag sign the footer. - Switch to
encrypted-footer.parquet. No column can be located. The structure view shows a plaintextFileCryptoMetaDataand one encrypted module where theFileMetaDatashould be.
What a plaintext footer shows#
| A reader without keys | What | Detail |
|---|---|---|
| can see | The encryption algorithm | AES_GCM_V1 |
| can see | The footer’s signature, which only the footer key can check | 28 bytes |
| can see | The schema | order_id, country, email, amount_cents |
| can see | The number of rows | 12 |
| can see | The values of order_id | 1, 2, 3, 4, … |
| can see | The statistics of order_id | 1 to 12 |
| can see | The values of country | “UK”, “SE”, “UK”, “PL”, … |
| can see | The statistics of country | “PL” to “US” |
| can see | Which key protects email | pii |
| can see | Which key protects amount_cents | finance |
| cannot see | The values of email | column email is encrypted with its own key, and this reader has no keys |
| cannot see | The statistics of email | withheld from the plaintext footer |
| cannot see | The values of amount_cents | column amount_cents is encrypted with its own key, and this reader has no keys |
| cannot see | The statistics of amount_cents | withheld from the plaintext footer |
Computed by the reader from fixtures/plaintext-footer.parquet (2192 bytes).
With a plaintext footer, the file still starts and ends with PAR1, and a reader that knows
nothing of encryption reads the unencrypted columns as usual. The encrypted columns keep a
reduced ColumnMetaData in the footer: their type, encodings, codec, sizes and offsets, but no
statistics. The full metadata, statistics included, is itself an encrypted module inside the
footer. Readers without keys therefore cannot skip by those columns’ statistics, and cannot read
them at all.
The footer ends with a signature: a nonce and a GCM tag computed with the footer key. A reader with that key can check that nobody changed the footer. A reader without it cannot, and has to trust the footer as it is.
What an encrypted footer shows#
| A reader without keys | What | Detail |
|---|---|---|
| can see | That it is Parquet, with an encrypted footer | PARE |
| can see | The encryption algorithm | AES_GCM_V1 |
| can see | The footer key’s name | footer |
| can see | The encrypted footer’s size | 954 bytes |
| cannot see | The schema | inside the encrypted footer |
| cannot see | The number of rows | inside the encrypted footer |
| cannot see | Where any column is | inside the encrypted footer |
| cannot see | Any statistics | inside the encrypted footer |
| cannot see | Any values, even of columns not encrypted | inside the encrypted footer |
Computed by the reader from fixtures/encrypted-footer.parquet (2123 bytes).
With an encrypted footer, the file starts and ends with PARE, and the footer is a small
plaintext FileCryptoMetaData, naming the algorithm and the footer key, followed by the real
FileMetaData as one module. Even the unencrypted columns cannot be read: their bytes are there
in plain form, but the footer that says where they are is not.
What still leaks#
Encryption hides values, not shape. From the steps, the tables above and the panel, a reader without keys still learns:
- the file’s size, and in a plaintext footer every column chunk’s size and offset;
- how many pages each encrypted column has, and how long each is. AES-GCM does not change a module’s length, so the ciphertext is exactly as long as the page, and page lengths can say something about the values;
- which columns are protected, and by which named keys.
The encrypted footer hides more of the shape, at the price of making the whole file unreadable without the footer key.
Building it#
The steps read the magic, the key metadata and one module by hand, and stopped where a key would be needed. The reader does the same for every module, in both footer modes, and says where it stops.
Walking modules#
A module is read from its length prefix alone, as the step read one, and an encrypted column chunk is a run of them:
def read_module(r: ByteReader) -> Module:
"""Read the module that starts at ``r``, from its length prefix alone."""
start = r.offset()
n = r.read_le_u32()
if n < NONCE_LEN + TAG_LEN:
why = f"a module at offset {start} claims {n} bytes, fewer than a nonce and a tag"
raise CryptoError(why)
_, body = r.read_bytes(n)
return Module(
span=Span(start, body.end),
length=Span(start, start + 4),
nonce=Span(body.start, body.start + NONCE_LEN),
ciphertext=Span(body.start + NONCE_LEN, body.end - TAG_LEN),
tag=Span(body.end - TAG_LEN, body.end),
)
pub fn read_module(r: &mut ByteReader) -> Result<Module, String> {
let start = r.offset();
let len = r.read_le_u32().map_err(|e| e.to_string())? as u64;
if len < NONCE_LEN + TAG_LEN {
return Err(format!(
"a module at offset {start} claims {len} bytes, fewer than a nonce and a tag"
));
}
let (_, body) = r.read_bytes(len as usize).map_err(|e| e.to_string())?;
Ok(Module {
span: Span::new(start, body.end),
length: Span::new(start, start + 4),
nonce: Span::new(body.start, body.start + NONCE_LEN),
ciphertext: Span::new(body.start + NONCE_LEN, body.end - TAG_LEN),
tag: Span::new(body.end - TAG_LEN, body.end),
})
}
An encrypted footer#
A file that ends with PARE has a footer in two parts. The reader decodes the first, the
plaintext FileCryptoMetaData, for the algorithm and the footer key’s metadata, and reads the
rest as one module, which must end where the footer does:
def encrypted_footer(file: bytes) -> EncryptedFooter:
"""Read an encrypted-footer file's footer, as far as a reader without its key can."""
size = len(file)
if size < 12 or file[:4] != MAGIC_ENCRYPTED or file[-4:] != MAGIC_ENCRYPTED:
why = "not an encrypted-footer file: it does not start and end with PARE"
raise CryptoError(why)
length = int.from_bytes(file[-8:-4], "little")
footer = footer_span(size, length)
r = ByteReader(file[footer.start : footer.end], footer.start)
try:
crypto_metadata = read_struct(r)
except ValueError as e:
raise CryptoError(f"FileCryptoMetaData: {e}") from e
s = as_struct(crypto_metadata)
algorithm = union_member(s.field(1).node) if s.field(1) else "UNKNOWN"
key = s.field(2)
key_metadata = key.node.value if key and isinstance(key.node.value, bytes) else None
module = read_module(r)
if not r.is_at_end():
left = footer.end - r.offset()
why = f"{left} bytes follow the encrypted FileMetaData inside the footer"
raise CryptoError(why)
return EncryptedFooter(footer, crypto_metadata, algorithm, key_metadata, module)
pub fn encrypted_footer(file: &[u8]) -> Result<EncryptedFooter, String> {
let n = file.len();
if n < 12 || file[..4] != MAGIC_ENCRYPTED || file[n - 4..] != MAGIC_ENCRYPTED {
let why = "not an encrypted-footer file: it does not start and end with PARE";
return Err(why.into());
}
let length = u32::from_le_bytes(file[n - 8..n - 4].try_into().unwrap());
let footer = crate::format::footer_span(n as u64, length).map_err(|e| e.to_string())?;
let bytes = &file[footer.start as usize..footer.end as usize];
let mut r = ByteReader::new(bytes, footer.start);
let crypto_metadata = read_struct(&mut r).map_err(|e| format!("FileCryptoMetaData: {e}"))?;
let s = crate::metadata::as_struct(&crypto_metadata).map_err(|e| e.to_string())?;
let algorithm = s
.field(1)
.map(|f| crate::metadata::union_member(&f.node))
.unwrap_or_else(|| "UNKNOWN".into());
let key_metadata = match s.field(2).map(|f| &f.node.value) {
Some(crate::thrift::Value::Binary(b)) => Some(b.clone()),
_ => None,
};
let module = read_module(&mut r)?;
if !r.is_at_end() {
return Err(format!(
"{} bytes follow the encrypted FileMetaData inside the footer",
footer.end - r.offset()
));
}
Ok(EncryptedFooter {
footer,
crypto_metadata,
algorithm,
key_metadata,
module,
})
}
A signed plaintext footer#
In the step that read one module, open_bytes decoded the plaintext footer with the reader’s
footer decoder, the metadata module. That decoder refuses bytes left over after the
FileMetaData: they mean the trailer’s length and the structure disagree. An encrypted file with
a plaintext footer leaves exactly a signature’s worth, and says it uses encryption:
# An encrypted file with a plaintext footer (ch13) signs it: the FileMetaData is followed by
# a 12-byte nonce and a 16-byte AES-GCM tag, and the footer length counts them.
footer_signature = None
if not r.is_at_end():
rest = Span(r.offset(), base + len(footer))
if s.field(8) is None or rest.length != FOOTER_SIGNATURE_LEN:
raise TrailingBytes(rest)
footer_signature = rest // An encrypted file with a plaintext footer (ch13) signs it: the FileMetaData is followed by
// a 12-byte nonce and a 16-byte AES-GCM tag, and the footer length counts them.
let mut footer_signature = None;
if !r.is_at_end() {
let rest = Span::new(r.offset(), base + footer.len() as u64);
if s.field(8).is_some() && rest.len() == FOOTER_SIGNATURE_LEN {
footer_signature = Some(rest);
} else {
return Err(MetadataError::TrailingBytes { span: rest });
}
}Refusing what it cannot read#
The column reader refuses an encrypted column by name, rather than decoding ciphertext as if it were a page, which is what a library without encryption does under Ask a library below:
def refuse_encrypted(chunk: ColumnChunk) -> None:
"""An encrypted column chunk's pages are encrypted modules (ch13). Without its key
they cannot be read, and the reader says so rather than decoding ciphertext."""
if chunk.crypto is not None:
key = "its own key" if chunk.crypto.with_column_key else "the footer key"
path = chunk.dotted_path()
why = f"column {path} is encrypted with {key}, and this reader has no keys"
raise ColumnError(why)
fn refuse_encrypted(chunk: &ColumnChunk) -> Result<(), ColumnError> {
match &chunk.crypto {
Some(c) => err(format!(
"column {} is encrypted with {}, and this reader has no keys",
chunk.dotted_path(),
if c.with_column_key {
"its own key"
} else {
"the footer key"
}
)),
None => Ok(()),
}
}
Checking it#
The fixtures were written by pyarrow with the keys, and the manifests describe them as pyarrow saw them with the keys. The tests hold the keyless reader to that description: the encrypted footer module is exactly the size pyarrow reports for the footer, the column chunks pyarrow found fill exactly the bytes before it, the unencrypted columns decode to pyarrow’s rows, and the encrypted columns’ modules tile their chunks.
python3 -m pytest python/tests -k encryptcargo test -p parquet-lab --test fixtures encryptEncryption uses a fresh nonce for every module and a fresh data key for every file, so these two fixtures cannot be regenerated byte for byte. They were written once; the generator’s check decrypts them with the published keys and confirms they still hold exactly the intended rows.
Ask a library#
At work a library reads these files, with keys from a key service. Without keys, pyarrow and the
parquet crate stop where the book’s reader stops, though not in the same way:
# pyarrow without keys: a plaintext footer opens, an encrypted one does not.
import pyarrow.parquet as pq
path = "fixtures/plaintext-footer.parquet"
metadata = pq.read_metadata(path)
print(f"plaintext-footer.parquet: {metadata.num_rows} rows")
for i in [0, 1]: # never 2 or 3: pyarrow stops Python on an encrypted column's metadata
chunk = metadata.row_group(0).column(i)
s = chunk.statistics
print(f" {chunk.path_in_schema}: statistics {s.min} to {s.max}")
for columns in [["order_id", "country"], ["email"]]: # try ["amount_cents"]
try:
table = pq.read_table(path, columns=columns)
print(f" read {columns}: {table.num_rows} rows")
except OSError as e:
print(f" read {columns}: {e}")
try:
pq.read_metadata("fixtures/encrypted-footer.parquet")
except OSError as e:
print(f"encrypted-footer.parquet: {e}")
//! The `parquet` crate without keys: a plaintext footer opens, an encrypted one does not.
use std::fs::File;
use parquet::file::reader::{FileReader, SerializedFileReader};
use parquet::file::statistics::Statistics;
use parquet::schema::types::Type;
/// A column's bounds, if the footer holds them where this reader can read them.
fn bounds(statistics: Option<&Statistics>) -> String {
match statistics {
Some(Statistics::Int64(s)) => {
format!("{} to {}", s.min_opt().unwrap(), s.max_opt().unwrap())
}
Some(Statistics::ByteArray(s)) => {
let (min, max) = (s.min_opt().unwrap(), s.max_opt().unwrap());
format!(
"{:?} to {:?}",
min.as_utf8().unwrap(),
max.as_utf8().unwrap()
)
}
_ => "none".to_string(),
}
}
fn main() {
let path = "fixtures/plaintext-footer.parquet";
let file = File::open(path).expect("run this from the repository's root");
let reader = SerializedFileReader::new(file).expect("a plaintext footer");
let metadata = reader.metadata();
let rows = metadata.file_metadata().num_rows();
println!("plaintext-footer.parquet: {rows} rows");
for chunk in metadata.row_group(0).columns() {
let name = chunk.column_path().string();
println!(" {name}: statistics {}", bounds(chunk.statistics()));
}
// Without its encryption feature the crate does not know email's pages are encrypted.
let fields = metadata.file_metadata().schema().get_fields();
let column = &fields[2]; // email; try 3, amount_cents
let projection = Type::group_type_builder("schema").with_fields(vec![column.clone()]);
let mut rows = reader
.get_row_iter(Some(projection.build().unwrap()))
.expect("rows");
match rows.next().expect("a row") {
Ok(row) => println!(" {}'s first row: {row}", column.name()),
Err(e) => println!(" {}'s first row: {e}", column.name()),
}
let file = File::open("fixtures/encrypted-footer.parquet").unwrap();
match SerializedFileReader::new(file) {
Ok(_) => println!("encrypted-footer.parquet: opened"),
Err(e) => println!("encrypted-footer.parquet: {e}"),
}
}
Both open the plaintext footer and count the rows. pyarrow gives the statistics of order_id and
country and reads those two columns. Asked for email, it refuses, because it cannot decrypt
the column’s metadata. The crate, built here without its encryption feature, finds no
statistics for the two encrypted columns, as the book’s reader found none. But it does not know
that email’s pages are encrypted: it reads the first module’s length prefix as the start of a
page header, and fails on a field the header lacks. The book’s reader refuses the column by name
before it reads a page. Neither library opens the encrypted footer without keys.
The Python step keeps away from one call. pyarrow’s metadata for an encrypted column,
metadata.row_group(0).column(2), needs that column’s key. Without the key, pyarrow does not
raise an error: it stops the whole Python process, and in the page that stops every step. A
reader with keys passes decryption properties from pyarrow.parquet.encryption, as the pyarrow
documentation under Where to go next shows. Ask for the other encrypted column,
amount_cents, instead of email: both libraries fail on it as they did on email.
In Python, the first run loads pyarrow into the page, a much larger download than the other
steps. In Rust the step needs the parquet crate, in walkthroughs/libraries, so it runs in a
Codespace or at a desk:
cargo run -q --manifest-path walkthroughs/libraries/Cargo.toml --bin encryption_with_a_libraryWhat this cannot tell you#
How to decrypt. This reader stops where the keys would be needed. Decrypting needs AES, GCM, the service that unwraps data keys, and the exact authenticated data for each module; the format specifies all of them, and this book does not implement them.
Whether to encrypt. Encryption protects files wherever they are copied. Access control on the storage protects them only where they are stored. Which risk matters is a question about the data, not the format.
How table formats use it. Engines that manage many files, ch14’s subject, decide which columns to encrypt and hold the keys; the files only carry the result.
Key takeaways#
- Parquet encrypts modules, not files. The footer and every page header and page are encrypted separately, so pages can still be read one at a time.
- A plaintext footer keeps the file readable without keys, except for the encrypted columns, whose values and statistics are hidden.
- An encrypted footer hides the whole layout. Without the footer key even unencrypted columns cannot be found.
- Files name keys; they never hold them. Data keys are wrapped by master keys that stay in a key service.
- Encryption hides values, not shape. Sizes, page counts and page lengths remain visible.
Problems#
Three, in exercises/python/modular_encryption.py, or in Rust in
exercises/src/modular_encryption.rs. The first two have tests. The third has none.
The editor below holds the Python stubs and keeps your edits in this browser. Run the tests runs the graders in the page, the same tests that run at a desk.
Here you can edit this chapter's problems and run their tests in your browser, in Python. It needs JavaScript. The same problems run at a desk: see Appendix A.
13.1 Find the modules. Split an encrypted column chunk into its modules by their length prefixes. The test compares your modules with the reader’s for both encrypted columns, and with generated chunks.
13.2 Which footer mode? Tell an unencrypted file, a plaintext footer and an encrypted footer apart. The test checks every fixture.
13.3 Your own threat model. No test: the data is yours. Pick a table with sensitive columns and decide which to encrypt, and whether the footer should be. List what a reader without keys would still learn in your choice, from the sizes, offsets and page counts this chapter showed. A good answer names one thing the plaintext footer reveals that matters for your data, and says whether that justifies an encrypted footer and the cost of making every reader hold a key.
For the Rust stubs, or to work at a desk, run the chapter’s graders with these commands. Add
-k problem_13_1 in Python, or problem_13_1 before the -- in Rust, to run one problem:
python3 -m pytest exercises/python/tests/test_modular_encryption.py --problemscargo test -p exercises --test modular_encryption -- --ignoredWhere to go next#
- Modular encryption is specified in the format repository’s Encryption.md.
- pyarrow’s encryption documentation shows how a real key service is plugged in.
- AES-GCM is specified in NIST Special Publication 800-38D.
- ch14 turns from one file to the many that make a table.
Built from commit c32f48e.