lance-format / lance-format/lance

Implement scalar-indexed joins on v2 merge_insert path

Open
#6,444 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

What to build

Implement the indexed join approach chosen in #6443 so that upserts with a
scalar index on the join key use the v2 path instead of falling back to v1.

Remove the scalar index fallback from can_use_create_plan. The use_index
parameter is retained as an escape hatch that controls whether the indexed
join node is used in the plan.

See milestone description for full PRD.

Acceptance criteria

  • Upserts with scalar-indexed join keys use the v2 path (verified via explain_plan)
  • Existing indexed merge insert tests pass unchanged
  • New explain_plan test confirms indexed join node appears in plan
  • use_index=false still works as escape hatch (bypasses index, uses full scan join)
  • Conflict detection (Bloom filter) works for indexed operations

Blocked by

  • #6443 (research spike must complete first)

User stories addressed

  • 3: indexed upserts use v2 path
  • 4: one fewer case on v1, moving toward single code path
  • 5: use_index retained as escape hatch

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the research spike in #6443 and the merge_insert planning path around can_use_create_plan. Trace explain_plan for scalar-indexed join keys, then verify the indexed join node, the use_index=false full-scan escape hatch, unchanged indexed merge insert tests, and Bloom-filter conflict detection. Done means all listed acceptance criteria pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.