opensearch-project / opensearch-project/OpenSearch
Completion suggestors consume a lot of on-heap FST memory
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Is your feature request related to a problem? Please describe.
OpenSearch cluster 2.5 ran into high JVMMP due to completion suggesters occupying large amount of FST on-heap store. This can also possibly grow with cat/indices or stats API loading this eagerly(FST is lazily loaded)
GET /_stats/completion
## For just one index
..
....
"autocomplete-num-01": {
"uuid": "hs2f122231daNerOIDX-V3g",
"primaries": {
"completion": {
"size_in_bytes": 3832664543
}
},
"total": {
"completion": {
"size_in_bytes": 11497993629
}
}
}
Describe the solution you'd like
Explore options to cap allocation of max FST memory. Wondering if we can move this off-heap?
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the completion suggesters and the _stats/completion entry point, then trace how the lazily loaded FST is accounted for in heap memory. Compare options for capping FST allocation or moving it off-heap, and define completion when the chosen approach limits memory growth without breaking completion statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100