apache / apache/lucene

FieldInfo#name contributes significantly to heap usage at scale [LUCENE-10676]

Open
#11,711 3 comments 0 reactions 0 assignees View on GitHub
affects-version:9.3 legacy-jira-label:heap legacy-jira-label:scalability legacy-jira-priority:Minor module:core/codecs type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We encountered an Elasticsearch user with high heap usage, a significant proportion of which was down to the contents of `FieldInfo#name`.

This user was certainly pushing some scalability boundaries: this single process had thousands of active Lucene indices, many with 10k+ fields, and many indices had hundreds of segments due to an excess of flushes, so in total they had an enormous number of `FieldInfo` instances. Still, the bulk of the heap usage was just field names, and the total number of distinct field names was fairly small. That's pretty common, especially for time-based data like logs. Some kind of interning or deduplication of these strings would have reduced their heap usage by many GBs.

Is there a way we could deduplicate these strings? Deduplicating them across segments within each index would already have helped, but ideally we'd like to deduplicate them across indices too.

---
Migrated from [LUCENE-10676](https://issues.apache.org/jira/browse/LUCENE-10676) by David Turner, 1 vote, updated Aug 09 2022
Environment:
```
Seen in Lucene 9.3.0 running on Linux using JDK18 but seems independent of environment.
```

Attachments: [image-2022-08-08-13-23-37-050.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10676/image-2022-08-08-13-23-37-050.png)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how FieldInfo#name strings are created and retained across segments and indices, using the reported Lucene 9.3.0 heap-usage scenario as context. Compare possible deduplication scopes and validate the chosen approach with measurements; done means field names use substantially less heap without changing their behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.