Avoid re-encoding dictionaries when writing to Parquet
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
### Discussed in https://github.com/apache/arrow-rs/discussions/8300
Originally posted by **duarten** September 9, 2025
It seems we always [re-encode dictionaries](https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/arrow_writer/byte_array.rs#L342) when writing row groups to Parquet.
Would it make sense to avoid this for cases where all/most dictionary values appear in all row groups?
Potentially related:
- https://github.com/apache/arrow-rs/pull/9520
- https://github.com/apache/arrow-rs/pull/2322
Contributor guide
Research direction
Start in parquet/src/arrow/arrow_writer/byte_array.rs around line 342, then read discussion 8300 and the related pull requests 9520 and 2322. Determine the intended approach for avoiding dictionary re-encoding and define completion around an agreed behavior for values shared across row groups; the issue does not specify tests or acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100