apache / apache/lucene

Reduce memory usage of skip reader

Open
#13,623 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Following up on #13576. Currently, we always use the size `Lucene99PostingsFormat.MAX_SKIP_LEVELS` to initialize the skip reader.

https://github.com/apache/lucene/blob/30c965ea575a5e75c2bf724a340aa690d82f1ec5/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99PostingsReader.java#L496-L501

We can take the max docs into consideration to calculate the maximum number of skip levels like we did in `MultiLevelSkipListWriter`.

https://github.com/apache/lucene/blob/30c965ea575a5e75c2bf724a340aa690d82f1ec5/lucene/core/src/java/org/apache/lucene/codecs/MultiLevelSkipListWriter.java#L75C8-L82

Contributor guide

Open the contributing guide

Research direction

Start in Lucene99PostingsReader.java around the skip-reader initialization linked in the issue, then compare the maximum-level calculation in MultiLevelSkipListWriter.java. Confirm that the reader's allocation accounts for the maximum document count rather than always using Lucene99PostingsFormat.MAX_SKIP_LEVELS, and verify the relevant Lucene tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.