apache / apache/lucene

Remove state from stored fields readers [LUCENE-9955]

Open
#10,994 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Because stored fields readers are cached per-segment per-thread, they can quickly start using lots of memory. E.g. an innocent 10kB buffer with 10 Lucene indices that have 100 segments each and are queried by 100 threads would eventually end up using 1GB of heap.

---
Migrated from [LUCENE-9955](https://issues.apache.org/jira/browse/LUCENE-9955) by Adrien Grand (@jpountz), updated Jul 19 2021
Pull requests: https://github.com/apache/lucene/pull/137

Contributor guide

Open the contributing guide

Research direction

Start by tracing how stored fields readers are cached per segment and thread, focusing on where their state is retained. Confirm that the change removes the accumulated per-reader state and prevents the memory growth described for multiple indices, segments, and threads; the migrated LUCENE-9955 issue and linked pull request provide additional context.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.