lance-format / lance-format/lance
Allow FTS indices to be used in filters
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We currently allow NGRAM indices to be used to speed up contains filters. NGRAM indices are limited to a very narrow range of tokenizers which should avoid false negatives from the index.
Inverted indices (FTS indices) on the other hand, could also be used to speed up contains filters, but might introduce false negatives (e.g. a query for bakin will not match a document with baking if stemming is turned on). However, some users may know they are not going to experience these false negatives (e.g. query string is whole words) or just wants to accept the chance of false negatives to speed up performance.
We could introduce a UDF contains_tokens which is similar to contains but allows an FTS / Inverted index to be used. This makes it possible for the user to opt-in to allowing these indices.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing contains filter handling and the code paths for NGRAM and inverted/FTS indices. Trace how a proposed contains_tokens UDF would opt into FTS index use, and define done as allowing whole-word or explicitly opted-in queries to use those indices while documenting the possible false negatives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100