apache / apache/arrow-rs

Use BitChunkIterator in Parquet Writer Level Computation

Open
#1,757 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

The level computation logic added in #1588 iterates through null bitmaps, checking each bit individually.

It is possible to iterate the set bits in a buffer more quickly, by instead reading u64 chunks, and counting leading zeros. This filter kernel's `IndexIterator` currently uses this trick, along with the parquet reader with `iter_set_bits_rev`.

**Describe the solution you'd like**

* Extract this bit mask iteration logic so that it can be reused
* Update the parquet writer to use this trick

**Describe alternatives you've considered**

We could not do this

**Additional Context**

Similar to https://github.com/apache/arrow-rs/issues/1600

Contributor guide

Open the contributing guide

Research direction

Start by comparing the filter kernel's IndexIterator and the Parquet reader's iter_set_bits_rev with the Parquet writer's level computation logic. Extract the shared set-bit iteration behavior so it can be reused, then update the writer to use u64 chunks and leading-zero counting. Done means the writer uses the reusable iterator and its existing behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.