lance-format / lance-format/lance
refactor(index): centralize data-overlay invalidation masking across index query paths
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Follow-up from #7549 (approved). Overlay/invalidation masking works today but is pieced together across many places, which makes it easy to get subtly wrong — e.g. the address↔row-id domain composition had to be threaded to every masking site to be correct under stable row ids.
The concept currently spans:
- 3 granularities: row-offset (
overlay_exclusion_offsets), fragment, and FTS segment. - 4 per-index-type scanner code paths (vector ANN, scalar V1
MaterializeIndexExec, scalar V2FilteredRead, FTS) — each computes stale rows, builds a block, threads it into the node, and assembles a separate flat-scan + take "re-add" path. - 4 node-side representations:
RowAddrMaskblock AND-ed intoDatasetPreFilter.final_mask(ANN + scalar V1) vs. fragment subtraction (EvaluatedIndex::without_fragments, scalar V2) vs. segment selection (FTS).
Proposal: a single OverlayInvalidation component per index that (a) computes stale rows once in one canonical domain, (b) exposes a uniform {block, re-add} pair, and (c) does the address↔row-id translation in exactly one place. That collapses the per-type scanner orchestration + the node-side knobs and makes the domain-composition bug impossible by construction.
Also fold in the DatasetPreFilter rename — it now assembles the full index-result mask at plan time (deletions + filter + overlay block), so "PreFilter" is misleading (noted in review).
Note: deletions (dropped everywhere; computed/applied by the nodes) and invalidations (moved index→brute-force; computed by the scanner) legitimately use different mechanisms — this is about consolidating the invalidation half, not merging the two concepts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the four scanner paths named in the issue: vector ANN, scalar V1 MaterializeIndexExec, scalar V2 FilteredRead, and FTS. Trace how each computes stale rows, builds its mask, and assembles the flat-scan re-add path, then inspect DatasetPreFilter and the node-side representations. Done means invalidation exposes one uniform block/re-add interface with address↔row-id translation centralized, while deletion handling remains separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100