lance-format / lance-format/lance
bug: FTS corpus statistics depend on the query filter and index coverage
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
BM25 scores change when neither the data nor the query changes, because the corpus the scores are measured against is derived partly from the rows a particular query happens to read.
Two triggers with the same root cause.
1. The user filter moves the corpus
The flat side of a partially indexed plan is read through filtered_read(filter_plan, ...), and initialize_scorer folds counted_input.num_rows() from that already-filtered scan into num_docs. Filtered-out tail rows are therefore missing from the corpus, while the index statistics still count filtered-out indexed rows.
2. Index coverage moves the corpus
A fully covered fragment selection is scored against index-only statistics, even when unindexed fragments exist outside the selection. Once optimize_indices() folds those fragments in, the same rows score differently.
Reproduce
Two indexed rows (title = alpha, beta), ten alpha rows appended unindexed, limit=1. Run the query, then optimize_indices(), then run it again. No searchable data changed:
| scenario | before | after |
|---|---|---|
prefilter(true), filter id < 2 |
[(1, 0.6931472)] |
[(1, 2.1594841)] |
with_fragments([0]) |
[(1, 0.6931472)] |
[(1, 2.1594841)] |
Both hold for single-column match too, where the returned row changes as well, not just its score: [(0, 0.6931472)] → [(1, 2.1594841)].
Scope
Affects single-column match and combined_fields alike — plan_match_query / plan_flat_match_query / initialize_scorer and the FlatFieldStats / SharedFtsScorer path have the same shape.
The overlay-triggered form of this invariant is already fixed for combined_fields in #7905 (FlatScanFilter::AtEmission, pinned by test_fts_combined_fields_overlay_preserves_prefilter_corpus), since a same-value overlay must not move a ranking. The filter- and coverage-triggered forms remain, for both query types.
Fix
Build one corpus that is independent of the filter and of index coverage, before either child runs, and apply the user filter only to emitted candidates. Candidate selection can stay fragment-scoped; only corpus construction has to be global.
Found in review of #7905:
https://github.com/lance-format/lance/pull/7905#discussion_r3955108697 and
https://github.com/lance-format/lance/pull/7905#discussion_r3955108708
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at plan_match_query, plan_flat_match_query, and initialize_scorer, then trace the FlatFieldStats and SharedFtsScorer paths for both match and combined_fields. Reproduce the filter and index-coverage cases described in the issue, and compare them with test_fts_combined_fields_overlay_preserves_prefilter_corpus from #7905. Done means corpus statistics and returned scores or rows remain stable when applying a user filter or optimizing index coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 62/100