lance-format / lance-format/lance
Migrate away from Merge
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The Merge operation is defined very generically. In fact, it is defined too generically. It gives the final set of fragments, not the set of fragments being modified. As a result, a merge operation will always conflict with other fragment-modifying operations (e.g. rewrite). This means something using merge (like merge_insert) will trigger a conflict even when the rows being modified are distinct from the rows being compacted. On a busy table we expect that compaction is almost always running. This makes it very expensive to run merge_insert on a busy table.
However, if we instead specified a list of modified, removed, and added fragments, then we could have better conflict detection.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the Merge operation and merge_insert implementation, then trace how fragment conflicts are determined. Done means merge operations identify modified, removed, and added fragments so distinct compaction and merge_insert work can avoid unnecessary conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100