lance-format / lance-format/lance
Auto-conflict resolution for compact and reindex
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
If one process re-indexes while another compacts, and then re-indexing commits first, users can get errors like:
Invalid compaction plan refers to index df68f95c-e005-4c61-ad9d-0e74710c355d which does not exist
This comes from
There may be other places where we get conflicts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/dataset/transaction.rs at lines 2552-2563 and trace the re-index and compaction transaction paths involved in the reported conflict. Identify how concurrent commits are handled and check for other conflict cases in the surrounding transaction logic. Done means the described concurrent operation no longer produces an invalid compaction-plan error and relevant conflict cases are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100