Consider using `try_for_each_valid_idx` for batches with mostly null values.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Most kernels evaluate the operations on all values regardless of whether it is null or not.
However, if the batch consists of mostly nulls, it may be faster to skip the null values. We can utilize the `try_for_each_valid_idx` function added recently by @tustvold
**Describe the solution you'd like**
At a certain threshold fraction of null values (based on benchmarking) we can use the `try_for_each_valid_idx` approach used by the checked variants of the kernels to speed up operations on batches with mostly nulls.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start by locating the recently added `try_for_each_valid_idx` function and the checked kernel variants that already use it. Benchmark batches with different null fractions, then determine a threshold and verify that kernels switch approaches for mostly-null batches without changing results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100