CharTermAttributeImpl.java's termBuffer caused heap out of memory.
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Relate to https://github.com/apache/lucene/issues/2934.
In Elasticsearch logging scenarios, a large message field may occasionally be processed for tokenization. This would expand `termBuffer` in `CharTermAttributeImpl`. Since there is no mechanism to shrink this buffer back, its memory footprint grows continuously as more indices would be added to the cluster, finally caused JVM OOM.
Here is the heap dump analysis result:
Single `TermAttributeImpl` used ~15MB buffer, total analyzer term buffer up to ~500MB.
### Version and environment details
Lucene version: 8.9.0
Single ES node: 32C 128G, 64GB heap.
Number of ES indices of the node: 700+ (MapperSerivce/Analyzer)
Contributor guide
Research direction
Start by reading CharTermAttributeImpl.java and the related Lucene issue 2934 to understand how termBuffer grows during tokenization. Use the reported large-field Elasticsearch scenario and heap-dump measurements as reproduction context; done means the buffer no longer retains an unbounded memory footprint after processing large terms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100