lance-format / lance-format/lance
Use bloom filters to determine IVF partitions when prefilter + scalar index
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
When there is a very restrictive prefilter we risk picking the wrong IVF partitions and not finding the correct results. One potential fix for this could be to keep a bloom filter of the row ids contained in each IVF partition. Given our prefilter results we can then eliminate partitions from consideration if none of the allowed ids are contained in the bloom filter.
This is not a sure thing and I am mostly jotting this down as an idea. There are still scenarios where this would not be helpful. For example, if the prefilter is very restrictive but the filtered results are very evenly distributed across the IVF partitions then this will not help much (we'd pretty much need to search all of the partitions no matter what). Other approaches, such as some kind of filter-aware IVF training, might be more effective.
Another potential solution would be to fallback to KNN if we find the prefilter is very selective.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named; start by locating IVF partition selection, scalar-index prefiltering, and the KNN fallback paths. Compare the proposed bloom-filter approach with fallback behavior, and define completion as preserving correct results for restrictive prefilters while avoiding unnecessary partition searches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100