apache / apache/arrow-rs

Speed up `BitIndexIterator` by unrolling

Open
#6,696 0 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
`BitIndexIterator` is critical to performance of lots of kernels (filter / take and anything dealing with nulls).

This can be made faster

**Describe the solution you'd like**

Implement optimization described here:
https://lemire.me/blog/2019/05/03/really-fast-bitset-decoding-for-average-densities/

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating `BitIndexIterator` in the arrow-rs source and read the linked Lemire article on fast bitset decoding. Compare the current iterator with the described unrolling approach, then verify that the change improves performance for the kernels that depend on it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.