lance-format / lance-format/lance

Research spike: indexed join strategy for v2 merge_insert

Open
#6,443 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

What to build

Time-boxed investigation to decide the approach for supporting scalar-indexed
joins in the v2 merge_insert path. Two approaches exist in the tracker:

  • ScalarIndexJoinExec (#3480)
  • Finger search (#4648)
Key questions
  • Can the existing v1 indexed join logic be wrapped as a DataFusion exec node?
  • Is a custom ScalarIndexJoinExec the right abstraction, or can we use
    DataFusion's built-in join with a custom table provider that reads from the
    index?
  • Does finger search apply as an optimization within whatever node we choose?

See milestone description for full PRD.

Acceptance criteria

  • Written design describing the chosen approach
  • #3480 description updated with the chosen design (or a new issue created if the approach diverges)
  • Key trade-offs documented (complexity, performance, reuse of existing code)

Blocked by

None - can start immediately.

User stories addressed

  • 3: indexed upserts use v2 path (design phase)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the milestone PRD, the existing v1 indexed join logic, and the designs in #3480 and #4648. Compare the possible DataFusion execution-node and table-provider approaches, including whether finger search fits, then document the chosen design, trade-offs, and update #3480 or create a replacement issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.