Remove state from stored fields readers [LUCENE-9955]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Because stored fields readers are cached per-segment per-thread, they can quickly start using lots of memory. E.g. an innocent 10kB buffer with 10 Lucene indices that have 100 segments each and are queried by 100 threads would eventually end up using 1GB of heap.
---
Migrated from [LUCENE-9955](https://issues.apache.org/jira/browse/LUCENE-9955) by Adrien Grand (@jpountz), updated Jul 19 2021
Pull requests: https://github.com/apache/lucene/pull/137
Contributor guide
Research direction
Start by tracing how stored fields readers are cached per segment and thread, focusing on where their state is retained. Confirm that the change removes the accumulated per-reader state and prevents the memory growth described for multiple indices, segments, and threads; the migrated LUCENE-9955 issue and linked pull request provide additional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100