lance-format / lance-format/lance
Potential index bloat when reampping with entirely deleted fragments
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
If an entire fragment is deleted then it is not included as part of the HashMap<u64, Option<u64>> that is passed to the index during remapping. As a result, the rows for that fragment are left behind in the index. This not only makes indexes larger but it wastes time because the index search will return matches that need to be filtered out. However, this does not lead to any correctness issues that I am aware of.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the remapping path that passes HashMap<u64, Option> to the index and inspect how completely deleted fragments are handled. Reproduce or add a regression test showing whether rows from such fragments remain in the index, then verify the index no longer returns those stale rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100