lance-format / lance-format/lance
Support phrase queries on massive datasets (possibly via new search strategies)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
In order to perform phrase queries we require that the FTS index be trained with position information. Unfortunately, this leads to huge indexes. For example, over 100GB on a 100M row sample of the fineweb dataset. The corresponding index without position was 6GB. These large indexes cannot be loaded into memory and so queries are often cold. This is fine for many use cases but for high scale use cases that want predictable latency this isn't workable. I fear this will also be unworkable on 1B+ row datasets because the majority of the index will be cold in that case.
We could investigate why the position information is so large and see if there is any way we can reduce it but I think some of this investigation has already been done and feel like any obvious fixes would have been found but maybe there is some radically different way of storing information that avoids this problem?
We could also investigate alternative search strategies. For example, in the default Postgres search index a phrase query turns into a match query against the index and then a phrase query is applied as an in-memory filter. In-memory FTS is expensive but perhaps there is some way we could specifically speed up the application of phrase queries (e.g. we don't have to tokenize?) Also, this would be a very useful fallback (with a warning potentially being logged that position information could help) on indexes that don't have position information (today we just error if there is a phrase query).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the FTS index handling for position information and the current phrase-query path, then compare the 100GB positioned index with the 6GB index without positions on the fineweb sample. Investigate the existing position-storage work and the PostgreSQL match-plus-in-memory-filter strategy; done means a defined storage or query strategy backed by predictable-latency benchmarks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100