Add blessed way to write Parquet row groups with different `WriterProperties`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
I'm converting between file formats in [silk-chiffon](https://github.com/acuitymd/silk-chiffon) and I'm working on making the Parquet files we write as optimal as possible.
To achieve that I need to buffer an entire row group in memory, perform cardinality analysis on it, then based on that decide whether it's worth using dictionary encoding and/or bloom filters (and the NDV for the latter!). This is how DuckDB writes out Parquet and it leads to some very tidy and fast Parquet files.
Doing this in the Parquet crate requires that I be able to use a different `WriterProperties` value per row group, and as far as I can tell the only to do that is the deprecated [`get_column_writers`](https://github.com/apache/arrow-rs/blob/d60b1cd7d60007382bf84c58ac7ba2626d887e19/parquet/src/arrow/arrow_writer/mod.rs#L427-L438).
So the request is this: can we add a blessed way to do this? If we were to do that, and if someone were want to contribute it back (👀), where would that fit into the API?
Thanks so much!
P.S. It was originally deprecated in https://github.com/apache/arrow-rs/pull/8582
Contributor guide
Research direction
Start in parquet/src/arrow/arrow_writer/mod.rs, especially the deprecated get_column_writers API referenced by the issue. Read the surrounding Arrow writer implementation and the deprecation context from pull request 8582, then determine the appropriate API boundary for per-row-group WriterProperties. Done means the crate provides a supported way to write row groups with differing properties.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100