Add memory usage check while constructing `OrdinalMap`
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
In our production environment, the `fielddata` memory usage can soar up to 12 GB, which makes the system highly prone to running out of memory. At present, we conduct memory checks only after the `OrdinalMap` has been constructed. However, this approach is merely a post - event measure. If multiple `OrdinalMaps` are built simultaneously or a single `OrdinalMap` is extremely large, leading to the exhaustion of memory, checking the memory usage at that point becomes completely ineffective.
Could we implement dynamic monitoring of `OrdinalMap` usage during its construction? For instance, we could check the memory usage every time 1024 terms are processed. This way, we can detect potential memory over - consumption issues in a more timely manner and take proactive measures to avoid system crashes caused by insufficient memory.
Contributor guide
Research direction
Start by locating OrdinalMap construction and the existing post-construction fielddata memory check in Lucene. Trace how terms are processed and determine how periodic checks, such as the suggested 1024-term interval, should respond to excessive usage. Done means memory is checked during construction and the behavior is covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100