lance-format / lance-format/lance
Invalid results from FTS or vector search after a data replacement
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
A DataReplacement operation can modify a column that is indexed. Today we will drop the modified fragment from the index's fragment bitmap. However, this does not avoid invalid results because the fragment bitmap might cover other fragments included in the search.
For example:
- Write fragments 1 & 2.
- Create index covering fragments 1 & 2.
- Update fragment 1 in-place with DataReplacement. Fragment 1 is removed from the index's bitmap.
- Search whole dataset
The search will target fragments 1 & 2. It will search the index because it covers fragment 2. It will search fragment 1 because it is unindexed. The search results from the index search will include fragment 1 rows even though it should not.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing DataReplacement handling and the FTS/vector search code that uses the index fragment bitmap. Reproduce the sequence of writing fragments 1 and 2, indexing them, replacing fragment 1, and searching the full dataset. Done means searches no longer return replaced fragment rows from the index while still covering valid indexed fragments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100