apache / apache/lucene

Paging with SortingMergePolicy and EarlyTerminatingSortingCollector [LUCENE-7255]

Open
#8,310 7 comments 0 reactions 0 assignees View on GitHub
affects-version:5.3 affects-version:5.4 affects-version:5.5 affects-version:6.0 legacy-jira-label:EarlyTerminatingSortingCollector legacy-jira-label:pagination legacy-jira-label:paging legacy-jira-label:searchafter legacy-jira-label:sortingmergepolicy legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

`EarlyTerminatingSortingCollector` seems to don't work when used with a `TopDocsCollector` searching for documents after a certain `FieldDoc`. That is, it can't be used for paging. The following code allows to reproduce the problem:

```Java
// Sort to be used both with merge policy and queries
Sort sort = new Sort(new SortedNumericSortField(FIELD_NAME, SortField.Type.INT));

// Create directory
RAMDirectory directory = new RAMDirectory();

// Setup merge policy
TieredMergePolicy tieredMergePolicy = new TieredMergePolicy();
SortingMergePolicy sortingMergePolicy = new SortingMergePolicy(tieredMergePolicy, sort);

// Setup index writer
IndexWriterConfig indexWriterConfig = new IndexWriterConfig(new SimpleAnalyzer());
indexWriterConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
indexWriterConfig.setMergePolicy(sortingMergePolicy);
IndexWriter indexWriter = new IndexWriter(directory, indexWriterConfig);

// Index values
for (int i = 1; i <= 1000; i++) {
Document document = new Document();
document.add(new NumericDocValuesField(FIELD_NAME, i));
indexWriter.addDocument(document);
}

// Force index merge to ensure early termination
indexWriter.forceMerge(1, true);
indexWriter.commit();

// Create index searcher
IndexReader reader = DirectoryReader.open(directory);
IndexSearcher searcher = new IndexSearcher(reader);

// Paginated read
int pageSize = 10;
FieldDoc pageStart = null;
while (true) {

logger.info("Collecting page starting at: {}", pageStart);

Query query = new MatchAllDocsQuery();

TopDocsCollector tfc = TopFieldCollector.create(sort, pageSize, pageStart, true, false, false);
EarlyTerminatingSortingCollector collector = new EarlyTerminatingSortingCollector(tfc, sort, pageSize, sort);
searcher.search(query, collector);
ScoreDoc[] scoreDocs = tfc.topDocs().scoreDocs;
for (ScoreDoc scoreDoc : scoreDocs) {
pageStart = (FieldDoc) scoreDoc;
logger.info("FOUND {}", scoreDoc);
}

logger.info("Terminated early: {}", collector.terminatedEarly());

if (scoreDocs.length < pageSize) break;
}

// Close
reader.close();
indexWriter.close();
directory.close();
```

The query for the second page doesn't return any results. However, it gets the expected results when if we don't wrap the `TopFieldCollector` with the `EarlyTerminatingSortingCollector`.

---
Migrated from [LUCENE-7255](https://issues.apache.org/jira/browse/LUCENE-7255) by Andres de la Peña, 1 vote, updated Apr 28 2016
Attachments: [LUCENE-7255_v0.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7255/LUCENE-7255_v0.diff)

Contributor guide

Open the contributing guide

Research direction

Start with the EarlyTerminatingSortingCollector and TopFieldCollector entry points, using the reproduction in the issue to compare the wrapped and unwrapped searches. Check why the second page returns no results after the first page's FieldDoc, and consider the issue done when paginated results match the unwrapped collector while early termination remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.