apache / apache/lucene

Hidden assumption that return value of IndexSearcher.slices is an array of continous sequential slices of the index [LUCENE-7861]

Open
#8,912 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.0 affects-version:6.5.1 legacy-jira-priority:Major module:core/search type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The IndexSearcher-method

```java
protected LeafSlice[] slices(List leaves)
```

can be overwritten to customize how the index is searched with multipe threads. However, the IndexSearcher assumes the result is an ordered array of continuous slices of the index. If the result is "interleaved" or unordered, searchAfter may skip results.

The issue seems to be how searchAfter works vs how TopDocs.merge works:

searchAfter skips every document with a higher score than the "after" document. In case of equal scores, it uses the document id and skips every document with a <= document id (see PagingFieldCollector).

TopDocs.merge uses the score to determine which hits should be part of the merged TopDocs. In case of equal scores, it uses the shard index (this corresponds to the slices the IndexSearcher uses) to break ties (see ScoreMergeSortQueue.lessThan)

So if the shards are noncontinuous/unordered, searchAfter uses a different way of sorting the documents than TopDocs.merge, and therefore hits are skipped.

On the mailing list, Michael McCandless suggested either improving TopDocs.merge to optionally use the docID for tie breaking (optionally as apparently the docId is not always global for every call of TopDocs.merge) or at least documenting the requirement on the return value of IndexSearcher.slices().

In my use case (generating a fixed amount of slices of approximately equal size), the requirement of ordered slices will result in a less optimal result - but I am not sure whether this has a real impact on performance.

---
Migrated from [LUCENE-7861](https://issues.apache.org/jira/browse/LUCENE-7861) by Christoph Kaser

Contributor guide

Open the contributing guide

Research direction

Start with IndexSearcher.slices, PagingFieldCollector, TopDocs.merge, and ScoreMergeSortQueue to trace how slice ordering and document IDs break searchAfter pagination. Compare the two tie-breaking rules described in the issue; done means an agreed fix or documented requirement prevents skipped results for unordered or interleaved slices.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.