apache / apache/arrow-rs

Expose Parquet encoding/type compatibility validation API

Open
#10,964 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

While adding `--column-encoding` in datafusion-contrib/tpcgen-rs#405, we needed a way to reject incompatible encodings before generating data, without duplicating parquet's compatibility checks.

`WriterPropertiesBuilder::set_column_encoding` accepts pairs that can panic during encoder construction: `RLE` on `INT64` in `RleValueEncoder::new` (`ensure_phys_ty!`), or `DELTA_BINARY_PACKED` on `BYTE_ARRAY` in `DeltaBitPackEncoder::new`.

Can we expose a public API to validate an `Encoding` against a physical type or column descriptor, returning `ParquetError` for unsupported pairs? Writer/encoder construction should use the same check rather than panic, so downstream validation stays in sync with parquet.

Related: apache/arrow-rs#10643 requests the same early validation in `parquet-rewrite`; apache/arrow-rs#1276 covered rejecting incompatible encoding/type pairs during decoder construction.

Contributor guide

Open the contributing guide

Research direction

Start with WriterPropertiesBuilder::set_column_encoding and the named encoder constructors RleValueEncoder::new and DeltaBitPackEncoder::new to trace where incompatible pairs are currently detected. Define the public validation behavior and ensure writer/encoder construction uses the same check, returning ParquetError instead of panicking for unsupported encoding and physical-type pairs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.