[Parquet] Make default fallback encoding choice smarter
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Currently, the choice of default encoding for a non-dictionary data page is trivial.
It happens in two places:
1. in the `FallbackToPlainEncoding` function for columns for which dictionary encoding is attempted:
https://github.com/apache/arrow/blob/5718a2862b4254d8bf938912d8958837ac7313a5/cpp/src/parquet/column_writer.cc#L1567-L1580
2. in the `ColumnWriter::Make` factory function for columns for which dictionary encoding is not attempted:
https://github.com/apache/arrow/blob/5718a2862b4254d8bf938912d8958837ac7313a5/cpp/src/parquet/column_writer.cc#L2375-L2382
I'll note that parquet-mr does not limit dictionary encoding fallback to PLAIN, even for "v1" Parquet files:
https://github.com/apache/parquet-mr/blob/95b004c3df473e3ab0963dc5136934ce5235d5df/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java#L124-L139
We should probably consolidate the logic from the two functions above and make it more sophisticated, allowing the best encoding for the selected Parquet version.
Also related: https://github.com/apache/arrow/issues/38441
### Component(s)
C++, Parquet
Contributor guide
Assessment
This issue has not been assessed yet.