opensearch-project / opensearch-project/OpenSearch
[Improvement] TransportCleanupRepositoryAction.cleanupRepo takes too much of cluster manager memory
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Describe the bug
TransportCleanupRepositoryAction.cleanupRepo flow utilizes the cluster manager JVM very quickly. We call doGetRepositoryData method, which internally takes up 11% of overall CPU utilization as well as 6.7% of overall memory allocations. This increase in JVMMP can impact the node join flow, as there is always a need of more memory by Gateway allocator(today).
Note : there is already a bug opened in case this code flow gets stuck and make generic threadpool busy, there the idea is if we can change the threadpool. Ref : https://github.com/opensearch-project/OpenSearch/issues/1788 . Here we need to explore about the optimizations to code path, if possible.
To Reproduce
Steps to reproduce the behavior:
- Create a big cluster with 3-4K indices and huge mapping (175MB of cluster state)
- Take 300 snapshots.
- Start this API for cleanup of your snapshot repository
- Take async profiler alloc profile to see the usage.
Expected behavior
We should optimize the code path to see if there are ways to improve the overall utilization and use lesser footprint while executing this cleanup.
Plugins
N/A
Screenshots
Attaching screenshots of async profiler.
Additional context
Attaching actual profiler files here, change them to .html to view the flamegraph.
steady-state-WITH-cleanup-cpu-profile.txt
steady-state-without-cleanup-cpu-profile.txt
steady-state-without-cleanup-alloc-profile.txt
steady-state-WITH-cleanup-alloc-profile.txt
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with TransportCleanupRepositoryAction.cleanupRepo and its doGetRepositoryData flow. Reproduce the issue with thousands of indices, a large cluster state, and many snapshots, then run the referenced async-profiler allocation and CPU profiles. Done means identifying and implementing a supported optimization that reduces cleanup's cluster-manager memory and CPU footprint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100