apache / apache/arrow-rs

replace bit-at-a-time null bitmap filtering with word-level

Open
#11,060 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

When filtering arrays with a null bitmap, the current implementation walks the bitmap one bit at a time, one byte load and branch per row. For mid-selectivity filters (~50% kept) this loop dominates `filter_bits`. we should replace it with a word-level approach that reduces the null bitmap pass from `O(selected_count)` byte loads to `O(filter_len/64)` u64 loads.

Contributor guide

Open the contributing guide

Research direction

Locate filter_bits and trace how selected rows and the null bitmap are represented in its callers. Read the current bit-at-a-time loop first; done means preserving filtering behavior while reducing the null-bitmap pass to word-level u64 loads with O(filter_len/64) work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.