lance-format / lance-format/lance

Support phrase queries on massive datasets (possibly via new search strategies)

Open
#6,104 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

In order to perform phrase queries we require that the FTS index be trained with position information. Unfortunately, this leads to huge indexes. For example, over 100GB on a 100M row sample of the fineweb dataset. The corresponding index without position was 6GB. These large indexes cannot be loaded into memory and so queries are often cold. This is fine for many use cases but for high scale use cases that want predictable latency this isn't workable. I fear this will also be unworkable on 1B+ row datasets because the majority of the index will be cold in that case.

We could investigate why the position information is so large and see if there is any way we can reduce it but I think some of this investigation has already been done and feel like any obvious fixes would have been found but maybe there is some radically different way of storing information that avoids this problem?

We could also investigate alternative search strategies. For example, in the default Postgres search index a phrase query turns into a match query against the index and then a phrase query is applied as an in-memory filter. In-memory FTS is expensive but perhaps there is some way we could specifically speed up the application of phrase queries (e.g. we don't have to tokenize?) Also, this would be a very useful fallback (with a warning potentially being logged that position information could help) on indexes that don't have position information (today we just error if there is a phrase query).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the FTS index handling for position information and the current phrase-query path, then compare the 100GB positioned index with the 6GB index without positions on the fineweb sample. Investigate the existing position-storage work and the PostgreSQL match-plus-in-memory-filter strategy; done means a defined storage or query strategy backed by predictable-latency benchmarks.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.