opensearch-project / opensearch-project/OpenSearch

[Improvement] TransportCleanupRepositoryAction.cleanupRepo takes too much of cluster manager memory

Open
#7,516 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug distributed framework
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

Describe the bug
TransportCleanupRepositoryAction.cleanupRepo flow utilizes the cluster manager JVM very quickly. We call doGetRepositoryData method, which internally takes up 11% of overall CPU utilization as well as 6.7% of overall memory allocations. This increase in JVMMP can impact the node join flow, as there is always a need of more memory by Gateway allocator(today).

Note : there is already a bug opened in case this code flow gets stuck and make generic threadpool busy, there the idea is if we can change the threadpool. Ref : https://github.com/opensearch-project/OpenSearch/issues/1788 . Here we need to explore about the optimizations to code path, if possible.

To Reproduce
Steps to reproduce the behavior:

  1. Create a big cluster with 3-4K indices and huge mapping (175MB of cluster state)
  2. Take 300 snapshots.
  3. Start this API for cleanup of your snapshot repository
  4. Take async profiler alloc profile to see the usage.

Expected behavior
We should optimize the code path to see if there are ways to improve the overall utilization and use lesser footprint while executing this cleanup.

Plugins
N/A

Screenshots
Attaching screenshots of async profiler.
steady-state-without-cleanup-alloc-profile
steady-state-without-cleanup-cpu-profile

steady-state-WITH-cleanup-alloc-profile
steady-state-WITH-cleanup-cpu-profile

Additional context
Attaching actual profiler files here, change them to .html to view the flamegraph.

steady-state-WITH-cleanup-cpu-profile.txt
steady-state-without-cleanup-cpu-profile.txt
steady-state-without-cleanup-alloc-profile.txt
steady-state-WITH-cleanup-alloc-profile.txt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with TransportCleanupRepositoryAction.cleanupRepo and its doGetRepositoryData flow. Reproduce the issue with thousands of indices, a large cluster state, and many snapshots, then run the referenced async-profiler allocation and CPU profiles. Done means identifying and implementing a supported optimization that reduces cleanup's cluster-manager memory and CPU footprint.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.