apache / apache/lucene

Add memory usage check while constructing `OrdinalMap`

Open
#14,961 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

In our production environment, the `fielddata` memory usage can soar up to 12 GB, which makes the system highly prone to running out of memory. At present, we conduct memory checks only after the `OrdinalMap` has been constructed. However, this approach is merely a post - event measure. If multiple `OrdinalMaps` are built simultaneously or a single `OrdinalMap` is extremely large, leading to the exhaustion of memory, checking the memory usage at that point becomes completely ineffective.

Image

Could we implement dynamic monitoring of `OrdinalMap` usage during its construction? For instance, we could check the memory usage every time 1024 terms are processed. This way, we can detect potential memory over - consumption issues in a more timely manner and take proactive measures to avoid system crashes caused by insufficient memory.

Contributor guide

Open the contributing guide

Research direction

Start by locating OrdinalMap construction and the existing post-construction fielddata memory check in Lucene. Trace how terms are processed and determine how periodic checks, such as the suggested 1024-term interval, should respond to excessive usage. Done means memory is checked during construction and the behavior is covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.