lance-format / lance-format/lance

refactor(index): centralize data-overlay invalidation masking across index query paths

Open
#7,898 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index chore
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Follow-up from #7549 (approved). Overlay/invalidation masking works today but is pieced together across many places, which makes it easy to get subtly wrong — e.g. the address↔row-id domain composition had to be threaded to every masking site to be correct under stable row ids.

The concept currently spans:

  • 3 granularities: row-offset (overlay_exclusion_offsets), fragment, and FTS segment.
  • 4 per-index-type scanner code paths (vector ANN, scalar V1 MaterializeIndexExec, scalar V2 FilteredRead, FTS) — each computes stale rows, builds a block, threads it into the node, and assembles a separate flat-scan + take "re-add" path.
  • 4 node-side representations: RowAddrMask block AND-ed into DatasetPreFilter.final_mask (ANN + scalar V1) vs. fragment subtraction (EvaluatedIndex::without_fragments, scalar V2) vs. segment selection (FTS).

Proposal: a single OverlayInvalidation component per index that (a) computes stale rows once in one canonical domain, (b) exposes a uniform {block, re-add} pair, and (c) does the address↔row-id translation in exactly one place. That collapses the per-type scanner orchestration + the node-side knobs and makes the domain-composition bug impossible by construction.

Also fold in the DatasetPreFilter rename — it now assembles the full index-result mask at plan time (deletions + filter + overlay block), so "PreFilter" is misleading (noted in review).

Note: deletions (dropped everywhere; computed/applied by the nodes) and invalidations (moved index→brute-force; computed by the scanner) legitimately use different mechanisms — this is about consolidating the invalidation half, not merging the two concepts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the four scanner paths named in the issue: vector ANN, scalar V1 MaterializeIndexExec, scalar V2 FilteredRead, and FTS. Trace how each computes stale rows, builds its mask, and assembles the flat-scan re-add path, then inspect DatasetPreFilter and the node-side representations. Done means invalidation exposes one uniform block/re-add interface with address↔row-id translation centralized, while deletion handling remains separate.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.