lance-format / lance-format/lance

Allow FTS indices to be used in filters

Open
#3,855 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

We currently allow NGRAM indices to be used to speed up contains filters. NGRAM indices are limited to a very narrow range of tokenizers which should avoid false negatives from the index.

Inverted indices (FTS indices) on the other hand, could also be used to speed up contains filters, but might introduce false negatives (e.g. a query for bakin will not match a document with baking if stemming is turned on). However, some users may know they are not going to experience these false negatives (e.g. query string is whole words) or just wants to accept the chance of false negatives to speed up performance.

We could introduce a UDF contains_tokens which is similar to contains but allows an FTS / Inverted index to be used. This makes it possible for the user to opt-in to allowing these indices.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing contains filter handling and the code paths for NGRAM and inverted/FTS indices. Trace how a proposed contains_tokens UDF would opt into FTS index use, and define done as allowing whole-word or explicitly opted-in queries to use those indices while documenting the possible false negatives.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.