Expose Parquet encoding/type compatibility validation API
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
While adding `--column-encoding` in datafusion-contrib/tpcgen-rs#405, we needed a way to reject incompatible encodings before generating data, without duplicating parquet's compatibility checks.
`WriterPropertiesBuilder::set_column_encoding` accepts pairs that can panic during encoder construction: `RLE` on `INT64` in `RleValueEncoder::new` (`ensure_phys_ty!`), or `DELTA_BINARY_PACKED` on `BYTE_ARRAY` in `DeltaBitPackEncoder::new`.
Can we expose a public API to validate an `Encoding` against a physical type or column descriptor, returning `ParquetError` for unsupported pairs? Writer/encoder construction should use the same check rather than panic, so downstream validation stays in sync with parquet.
Related: apache/arrow-rs#10643 requests the same early validation in `parquet-rewrite`; apache/arrow-rs#1276 covered rejecting incompatible encoding/type pairs during decoder construction.
Contributor guide
Research direction
Start with WriterPropertiesBuilder::set_column_encoding and the named encoder constructors RleValueEncoder::new and DeltaBitPackEncoder::new to trace where incompatible pairs are currently detected. Define the public validation behavior and ensure writer/encoder construction uses the same check, returning ParquetError instead of panicking for unsupported encoding and physical-type pairs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100