lance-format / lance-format/lance
Simplify rules when vector/fts used as filter
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The filter_query method in Scanner allows for an FTS or vector search to be used as a filter. Currently there are branches for both prefilter and postfilter and in the FTS filter path there are branches for "match query" vs. "not match query". I find these confusing and inconsistent.
- If we perform a full text search with a vector pre-filter, and it is a match query, then we rerank the vector search results, removing rows where the score is 0.
- If we perform a full text search with a vector pre-filter, and it is not a match query, then we perform both an FTS search and a vector search and do an inner join on the results on _rowid.
- If we perform a vector search with an FTS pre-filter, then we do a flat KNN on the FTS results
- If we perform a full text search with a vector post-filter, then we rerank the results by KNN distance (and don't actually filter anything)
- If we perform a vector search with an FTS post-filter, then we remove all results that do not share a token with the query
It is confusing that a full text search with a vector prefilter becomes a vector search (and vice versa). It also seems like a vector search with an FTS prefilter is the same thing as an FTS search with a vector postfilter (and vice versa).
I propose we simplify as follows:
- Allow FTS and vector to be used as post-filters for any kind of query (even scans that are not a search)
- Define an FTS post-filter as a filter that reranks the output and (optionally) removes all rows where the BM25 score is below some threshold
- Define a vector search post-filter as a filter that reranks the output and (optionally) removes all rows where the vector search distance is above some threshold
- Do not allow FTS or vector to be used as prefilters
- Do not have different behavior for match query and non-match queries. If we can't support non-match queries in filter mode then just return a "not supported" error until we can.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the Scanner.filter_query method and tracing its existing FTS and vector prefilter and post-filter branches. Compare the current match-query and non-match behavior with the proposed unified post-filter rules. Done means the rules are simplified as described, unsupported non-match filter cases return a clear error, and the relevant behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100