[Parquet][C++] More elaborate dictionary fallback for Parquet 2.0
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
In our standard ( https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8 ) , the dict fallback will goes to PLAIN encoding.
But in many implementions, it says that Parquet 2.0 should support fallback to other types, in our code, there is some todo:
```
// Only PLAIN encoding is supported for fallback in V1
// TODO(majetideepak): Use user specified encoding for V2
if (dictionary_fallback) {
thrift_encodings.push_back(ToThrift(Encoding::PLAIN));
}
```
And, in parquet-mr and arrow-rs, they both support fallback to other types:
1. parquet-mr: https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java
2. arrow-rs: https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/mod.rs#L1028-L1040
So, should we support that?
### Component(s)
C++, Parquet
Contributor guide
Research direction
Start at the C++ Parquet code containing the dictionary_fallback TODO and the thrift_encodings handling shown in the issue. Compare the parquet-mr DefaultV2ValuesWriterFactory and arrow-rs column writer references to determine the intended Parquet 2.0 fallback behavior. Done means an agreed implementation for user-specified fallback encodings, with validation against the relevant Parquet encoding standard.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100