FieldInfo#name contributes significantly to heap usage at scale [LUCENE-10676]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
We encountered an Elasticsearch user with high heap usage, a significant proportion of which was down to the contents of `FieldInfo#name`.
This user was certainly pushing some scalability boundaries: this single process had thousands of active Lucene indices, many with 10k+ fields, and many indices had hundreds of segments due to an excess of flushes, so in total they had an enormous number of `FieldInfo` instances. Still, the bulk of the heap usage was just field names, and the total number of distinct field names was fairly small. That's pretty common, especially for time-based data like logs. Some kind of interning or deduplication of these strings would have reduced their heap usage by many GBs.
Is there a way we could deduplicate these strings? Deduplicating them across segments within each index would already have helped, but ideally we'd like to deduplicate them across indices too.
---
Migrated from [LUCENE-10676](https://issues.apache.org/jira/browse/LUCENE-10676) by David Turner, 1 vote, updated Aug 09 2022
Environment:
```
Seen in Lucene 9.3.0 running on Linux using JDK18 but seems independent of environment.
```
Attachments: [image-2022-08-08-13-23-37-050.png](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10676/image-2022-08-08-13-23-37-050.png)
Contributor guide
Research direction
Start by tracing how FieldInfo#name strings are created and retained across segments and indices, using the reported Lucene 9.3.0 heap-usage scenario as context. Compare possible deduplication scopes and validate the chosen approach with measurements; done means field names use substantially less heap without changing their behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100