Use BitChunkIterator in Parquet Writer Level Computation
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The level computation logic added in #1588 iterates through null bitmaps, checking each bit individually.
It is possible to iterate the set bits in a buffer more quickly, by instead reading u64 chunks, and counting leading zeros. This filter kernel's `IndexIterator` currently uses this trick, along with the parquet reader with `iter_set_bits_rev`.
**Describe the solution you'd like**
* Extract this bit mask iteration logic so that it can be reused
* Update the parquet writer to use this trick
**Describe alternatives you've considered**
We could not do this
**Additional Context**
Similar to https://github.com/apache/arrow-rs/issues/1600
Contributor guide
Research direction
Start by comparing the filter kernel's IndexIterator and the Parquet reader's iter_set_bits_rev with the Parquet writer's level computation logic. Extract the shared set-bit iteration behavior so it can be reused, then update the writer to use u64 chunks and leading-zero counting. Done means the writer uses the reusable iterator and its existing behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100