lance-format / lance-format/lance
Research spike: indexed join strategy for v2 merge_insert
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
What to build
Time-boxed investigation to decide the approach for supporting scalar-indexed
joins in the v2 merge_insert path. Two approaches exist in the tracker:
ScalarIndexJoinExec(#3480)- Finger search (#4648)
Key questions
- Can the existing v1 indexed join logic be wrapped as a DataFusion exec node?
- Is a custom
ScalarIndexJoinExecthe right abstraction, or can we use
DataFusion's built-in join with a custom table provider that reads from the
index? - Does finger search apply as an optimization within whatever node we choose?
See milestone description for full PRD.
Acceptance criteria
- Written design describing the chosen approach
- #3480 description updated with the chosen design (or a new issue created if the approach diverges)
- Key trade-offs documented (complexity, performance, reuse of existing code)
Blocked by
None - can start immediately.
User stories addressed
- 3: indexed upserts use v2 path (design phase)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the milestone PRD, the existing v1 indexed join logic, and the designs in #3480 and #4648. Compare the possible DataFusion execution-node and table-provider approaches, including whether finger search fits, then document the chosen design, trade-offs, and update #3480 or create a replacement issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100