lance-format / lance-format/lance

Invalid results from FTS or vector search after a data replacement

Open
#6,283 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

A DataReplacement operation can modify a column that is indexed. Today we will drop the modified fragment from the index's fragment bitmap. However, this does not avoid invalid results because the fragment bitmap might cover other fragments included in the search.

For example:

  1. Write fragments 1 & 2.
  2. Create index covering fragments 1 & 2.
  3. Update fragment 1 in-place with DataReplacement. Fragment 1 is removed from the index's bitmap.
  4. Search whole dataset

The search will target fragments 1 & 2. It will search the index because it covers fragment 2. It will search fragment 1 because it is unindexed. The search results from the index search will include fragment 1 rows even though it should not.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing DataReplacement handling and the FTS/vector search code that uses the index fragment bitmap. Reproduce the sequence of writing fragments 1 and 2, indexing them, replacing fragment 1, and searching the full dataset. Done means searches no longer return replaced fragment rows from the index while still covering valid indexed fragments.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.