apache / apache/lucene

[Bug] Postings force merge regression between Lucene 9.12 and Lucene 10.0

Open
#14,514 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This is fork of the issue https://github.com/apache/lucene/issues/14463 specific to postings format regression.

As part of 10.0 Lucene, default readAdvice changed to random and we noticed that termsIndex which uses default readAdvice regressed during force merge.

```
String termsName =
IndexFileNames.segmentFileName(segment, state.segmentSuffix, TERMS_EXTENSION);
termsIn = state.directory.openInput(termsName, state.context);
```

So there could be other implementations and file formats which has similar regressions but fixing the `termsIn` removed the 20% regression we saw between OpenSearch 2.19 [ Lucene 9.12.1] and OpenSearch 3.0 [Lucene 10.1].

So creating this issue to track postings related force merge regressions and fixes.

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing the postings implementations involved in force merge, especially the termsIndex path and the shown IndexFileNames.segmentFileName/openInput entry points. Compare behavior between Lucene 9.12 and 10.0, then use force-merge measurements to identify and verify any remaining postings-format regression fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.