apache / apache/lucene

Improve memory footprint of SortingCodecReader [LUCENE-9539]

Open
#10,579 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

SortingCodecReader is a very memory heavy since it needs to re-sort and load large parts of the index into memory. We can try to make it more efficient by using more compact internal data-structures, remove the caches it uses provided we define it's usage as a merge only reader wrapper. Ultimately we need to find a way to allow the reader or some other structure to minimize its heap memory. One way is to slice existing readers and merge them in multiple steps. There will be multiple steps towards a more useable version of this class.

---
Migrated from [LUCENE-9539](https://issues.apache.org/jira/browse/LUCENE-9539) by Simon Willnauer (@s1monw), updated May 22 2021
Pull requests: https://github.com/apache/lucene-solr/pull/1908, https://github.com/apache/lucene-solr/pull/1909, https://github.com/apache/lucene-solr/pull/1908, https://github.com/apache/lucene-solr/pull/1909

Contributor guide

Open the contributing guide

Research direction

Start by locating SortingCodecReader and reviewing the LUCENE-9539 context and pull requests 1908 and 1909. Determine which memory-heavy structures and merge-only assumptions are in scope; done means reducing the reader's heap usage while preserving its affected merge behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.