lance-format / lance-format/lance

Use bloom filters to determine IVF partitions when prefilter + scalar index

Open
#1,595 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

When there is a very restrictive prefilter we risk picking the wrong IVF partitions and not finding the correct results. One potential fix for this could be to keep a bloom filter of the row ids contained in each IVF partition. Given our prefilter results we can then eliminate partitions from consideration if none of the allowed ids are contained in the bloom filter.

This is not a sure thing and I am mostly jotting this down as an idea. There are still scenarios where this would not be helpful. For example, if the prefilter is very restrictive but the filtered results are very evenly distributed across the IVF partitions then this will not help much (we'd pretty much need to search all of the partitions no matter what). Other approaches, such as some kind of filter-aware IVF training, might be more effective.

Another potential solution would be to fallback to KNN if we find the prefilter is very selective.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named; start by locating IVF partition selection, scalar-index prefiltering, and the KNN fallback paths. Compare the proposed bloom-filter approach with fallback behavior, and define completion as preserving correct results for restrictive prefilters while avoiding unnecessary partition searches.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.