lance-format / lance-format/lance
By default, assert there are no duplicate row ids matched in merge insert
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Merge insert is often used to avoid duplicates. However, if the source data or target table have duplicates, we will increase the number of duplicates. This can create huge number of duplicates if merge insert is called many times.
Instead, we should detect if duplicate rows are created, and error before writing out. This could be turned off with a flag if a user desired, but I expect most users would want this assertion on.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the merge insert implementation and its existing tests. Reproduce cases where the source or target contains duplicate row IDs, then verify that duplicate creation is detected before writing and that the requested opt-out behavior is covered if implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100