lance-format / lance-format/lance

Simplify rules when vector/fts used as filter

Open
#6,076 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

The filter_query method in Scanner allows for an FTS or vector search to be used as a filter. Currently there are branches for both prefilter and postfilter and in the FTS filter path there are branches for "match query" vs. "not match query". I find these confusing and inconsistent.

  • If we perform a full text search with a vector pre-filter, and it is a match query, then we rerank the vector search results, removing rows where the score is 0.
  • If we perform a full text search with a vector pre-filter, and it is not a match query, then we perform both an FTS search and a vector search and do an inner join on the results on _rowid.
  • If we perform a vector search with an FTS pre-filter, then we do a flat KNN on the FTS results
  • If we perform a full text search with a vector post-filter, then we rerank the results by KNN distance (and don't actually filter anything)
  • If we perform a vector search with an FTS post-filter, then we remove all results that do not share a token with the query

It is confusing that a full text search with a vector prefilter becomes a vector search (and vice versa). It also seems like a vector search with an FTS prefilter is the same thing as an FTS search with a vector postfilter (and vice versa).

I propose we simplify as follows:

  • Allow FTS and vector to be used as post-filters for any kind of query (even scans that are not a search)
  • Define an FTS post-filter as a filter that reranks the output and (optionally) removes all rows where the BM25 score is below some threshold
  • Define a vector search post-filter as a filter that reranks the output and (optionally) removes all rows where the vector search distance is above some threshold
  • Do not allow FTS or vector to be used as prefilters
  • Do not have different behavior for match query and non-match queries. If we can't support non-match queries in filter mode then just return a "not supported" error until we can.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Scanner.filter_query method and tracing its existing FTS and vector prefilter and post-filter branches. Compare the current match-query and non-match behavior with the proposed unified post-filter rules. Done means the rules are simplified as described, unsupported non-match filter cases return a clear error, and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.