lance-format / lance-format/lance
Differentiate between NaN, infinite with ZoneMap filter
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The incoming ZoneMap filter uses the DataFusion accumulator to calculate min/max which follows total ordering
Say we have an array of [NaN, 0.0, 3.0, 2.0, 1.0, 5.0], the max will be set to NaN rather than 5.0 which is les than ideal. If a lot of zones have the NaN value, the zonemap filter will become inefficient.
We should further improve the categorization for these special values to achieve optimal performance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the incoming ZoneMap filter and the DataFusion accumulator used to calculate min/max. Reproduce the array containing NaN, then define and verify separate categorization for NaN and infinite values so zones can be filtered efficiently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100