apache / apache/arrow

[Parquet] Make default fallback encoding choice smarter

Open
#40,636 2 comments 0 reactions 1 assignee Claimed by @ClifHouck View on GitHub
Component: C++ Component: Parquet Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Currently, the choice of default encoding for a non-dictionary data page is trivial.
It happens in two places:
1. in the `FallbackToPlainEncoding` function for columns for which dictionary encoding is attempted:
https://github.com/apache/arrow/blob/5718a2862b4254d8bf938912d8958837ac7313a5/cpp/src/parquet/column_writer.cc#L1567-L1580
2. in the `ColumnWriter::Make` factory function for columns for which dictionary encoding is not attempted:
https://github.com/apache/arrow/blob/5718a2862b4254d8bf938912d8958837ac7313a5/cpp/src/parquet/column_writer.cc#L2375-L2382

I'll note that parquet-mr does not limit dictionary encoding fallback to PLAIN, even for "v1" Parquet files:
https://github.com/apache/parquet-mr/blob/95b004c3df473e3ab0963dc5136934ce5235d5df/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java#L124-L139

We should probably consolidate the logic from the two functions above and make it more sophisticated, allowing the best encoding for the selected Parquet version.

Also related: https://github.com/apache/arrow/issues/38441

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.