Bulk skip in RunLengthBitPackingHybridDecoder / DictionaryValuesReader
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
### Motivation
Make Hive leverage bulk skip when implementing probe decode for Parquet, similarly to https://issues.apache.org/jira/browse/HIVE-22731, which was about ORC.
### Problem
`ValuesReader.skip(int n)` ships with a naive default:
```java
public void skip(int n) {
for (int i = 0; i < n; i++) skip();
}
```
For dictionary-encoded columns (the common case), each `skip()` bottoms
out in `RunLengthBitPackingHybridDecoder.readInt()` — a mode switch,
array-index arithmetic, and a value the caller immediately discards.
Any filter-then-skip path (column-index row ranges, hash-join probe
filtering, runtime filters) pays this cost per skipped row.
### Proposal
1. Add `RunLengthBitPackingHybridDecoder.skipInts(int n)` — re-use
`readNext()` per run, then advance `currentCount` by
`min(n, currentCount)` instead of walking every value through
`readInt()`.
2. Override `skip(int)` on `DictionaryValuesReader` and
`RunLengthBitPackingHybridValuesReader` to call `decoder.skipInts(n)`.
### Component(s)
Core
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading RunLengthBitPackingHybridDecoder.readInt() and readNext(), then inspect skip(int) in DictionaryValuesReader and RunLengthBitPackingHybridValuesReader. Confirm how dictionary-encoded values are decoded and identify the relevant existing tests or test entry points. Done means bulk skipping uses the decoder path in all three readers while preserving the number of values skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100