TestTopDocsMerge Is Broken [LUCENE-8824]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Investigating a test failure post-LUCENE-8757, I realized that TestTopDocsMerge takes a non-obvious invariant on the fact that the number of Collectors involved in the merge will be equal to the number of LeafReaderContexts originally present. This is propagated in the corresponding ScoreDocs's shardIndex fields, which can lead to subtle issues since shardIndex is used for tie-breaking in the priority queue used during the merge. I believe that this is a dangerous and unnecessary dependency to take since the IndexSearcher#slices method does not advertise any such guarantees.
The underlying assumption worked well in the past since the default slice allocation algorithm always allocated a slice per segment, thus guaranteeing that the number of Collectors (== number of Slices) will be equal to the number of Leaf Contexts. With 8757, this is no longer true.
I propose a rewrite of the test, where ShardSearcher is allowed to take a LeafSlice instance and can internally do a sequential search on the leaf contexts of the passed in the slice. This will allow TestTopDocsMerge to create N subsearchers where N is equal to the number of slices used by the IndexSearcher being compared to.
---
Migrated from [LUCENE-8824](https://issues.apache.org/jira/browse/LUCENE-8824) by Atri Sharma (@atris)
Contributor guide
Research direction
Start with the TestTopDocsMerge test and the ShardSearcher and LeafSlice references described in the issue; review the slice behavior introduced by LUCENE-8757 and the IndexSearcher#slices contract. The test is done when it no longer assumes one collector per LeafReaderContext, instead matching the IndexSearcher slice count while preserving correct merge and tie-breaking behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100