Improve memory footprint of SortingCodecReader [LUCENE-9539]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
SortingCodecReader is a very memory heavy since it needs to re-sort and load large parts of the index into memory. We can try to make it more efficient by using more compact internal data-structures, remove the caches it uses provided we define it's usage as a merge only reader wrapper. Ultimately we need to find a way to allow the reader or some other structure to minimize its heap memory. One way is to slice existing readers and merge them in multiple steps. There will be multiple steps towards a more useable version of this class.
---
Migrated from [LUCENE-9539](https://issues.apache.org/jira/browse/LUCENE-9539) by Simon Willnauer (@s1monw), updated May 22 2021
Pull requests: https://github.com/apache/lucene-solr/pull/1908, https://github.com/apache/lucene-solr/pull/1909, https://github.com/apache/lucene-solr/pull/1908, https://github.com/apache/lucene-solr/pull/1909
Contributor guide
Research direction
Start by locating SortingCodecReader and reviewing the LUCENE-9539 context and pull requests 1908 and 1909. Determine which memory-heavy structures and merge-only assumptions are in scope; done means reducing the reader's heap usage while preserving its affected merge behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100