apache / apache/arrow

[Parquet][C++] More elaborate dictionary fallback for Parquet 2.0

Open
#15,165 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

In our standard ( https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8 ) , the dict fallback will goes to PLAIN encoding.

But in many implementions, it says that Parquet 2.0 should support fallback to other types, in our code, there is some todo:

```
// Only PLAIN encoding is supported for fallback in V1
// TODO(majetideepak): Use user specified encoding for V2
if (dictionary_fallback) {
thrift_encodings.push_back(ToThrift(Encoding::PLAIN));
}
```

And, in parquet-mr and arrow-rs, they both support fallback to other types:
1. parquet-mr: https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV2ValuesWriterFactory.java
2. arrow-rs: https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/mod.rs#L1028-L1040

So, should we support that?

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Research direction

Start at the C++ Parquet code containing the dictionary_fallback TODO and the thrift_encodings handling shown in the issue. Compare the parquet-mr DefaultV2ValuesWriterFactory and arrow-rs column writer references to determine the intended Parquet 2.0 fallback behavior. Done means an agreed implementation for user-specified fallback encodings, with validation against the relevant Parquet encoding standard.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.