Paging with SortingMergePolicy and EarlyTerminatingSortingCollector [LUCENE-7255]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
`EarlyTerminatingSortingCollector` seems to don't work when used with a `TopDocsCollector` searching for documents after a certain `FieldDoc`. That is, it can't be used for paging. The following code allows to reproduce the problem:
```Java
// Sort to be used both with merge policy and queries
Sort sort = new Sort(new SortedNumericSortField(FIELD_NAME, SortField.Type.INT));
// Create directory
RAMDirectory directory = new RAMDirectory();
// Setup merge policy
TieredMergePolicy tieredMergePolicy = new TieredMergePolicy();
SortingMergePolicy sortingMergePolicy = new SortingMergePolicy(tieredMergePolicy, sort);
// Setup index writer
IndexWriterConfig indexWriterConfig = new IndexWriterConfig(new SimpleAnalyzer());
indexWriterConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
indexWriterConfig.setMergePolicy(sortingMergePolicy);
IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);
// Index values
for (int i = 1; i <= 1000; i++) {
Document document = new Document();
document.add(new NumericDocValuesField(FIELD_NAME, i));
indexWriter.addDocument(document);
}
// Force index merge to ensure early termination
indexWriter.forceMerge(1, true);
indexWriter.commit();
// Create index searcher
IndexReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);
// Paginated read
int pageSize = 10;
FieldDoc pageStart = null;
while (true) {
logger.info("Collecting page starting at: {}", pageStart);
Query query = new MatchAllDocsQuery();
TopDocsCollector tfc = TopFieldCollector.create(sort, pageSize, pageStart, true, false, false);
EarlyTerminatingSortingCollector collector = new EarlyTerminatingSortingCollector(tfc, sort, pageSize, sort);
searcher.search(query, collector);
ScoreDoc[] scoreDocs = tfc.topDocs().scoreDocs;
for (ScoreDoc scoreDoc : scoreDocs) {
pageStart = (FieldDoc) scoreDoc;
logger.info("FOUND {}", scoreDoc);
}
logger.info("Terminated early: {}", collector.terminatedEarly());
if (scoreDocs.length < pageSize) break;
}
// Close
reader.close();
indexWriter.close();
directory.close();
```
The query for the second page doesn't return any results. However, it gets the expected results when if we don't wrap the `TopFieldCollector` with the `EarlyTerminatingSortingCollector`.
---
Migrated from [LUCENE-7255](https://issues.apache.org/jira/browse/LUCENE-7255) by Andres de la Peña, 1 vote, updated Apr 28 2016
Attachments: [LUCENE-7255_v0.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7255/LUCENE-7255_v0.diff)
Contributor guide
Research direction
Start with the EarlyTerminatingSortingCollector and TopFieldCollector entry points, using the reproduction in the issue to compare the wrapped and unwrapped searches. Check why the second page returns no results after the first page's FieldDoc, and consider the issue done when paginated results match the unwrapped collector while early termination remains supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100