Improve the RLE encoding for Parquet Dictionary IDs
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
The IDs of Parquet Dictionary encoding is using `RunLengthBitPackingHybridEncoder`.
RunLengthBitPackingHybridEncoder handles encoding with `repeat` and `bitpacking`, we should improve it with the method likes `DeltaBinaryPackingWriter`
**Reporter**: [Dapeng Sun](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dapengsun) / @sundapeng
**Note**: *This issue was originally created as [PARQUET-1059](https://issues.apache.org/jira/browse/PARQUET-1059). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating RunLengthBitPackingHybridEncoder and DeltaBinaryPackingWriter in the Java sources, then read their encoding and test coverage. Determine the intended improved handling for Parquet Dictionary IDs and define completion through encoding correctness and performance tests; the issue does not name specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100