hashicorp / hashicorp/memberlist

Question: high CPU usage for a large cluster

Open
#243 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4.1k
Forks
478
Avg merge
15h 58m
Merged PRs (30d)
3

Description

I'm evaluating a memberlist based discovery on a large cluster (~ 2500 nodes).

The problem is that the prototype continuously consumes multiple (2-3) cores on each node. However, it didn't happen for smaller installations (~100 nodes).
Is this a normal cpu usage for a large cluster or some configuration issue?

I use DefaultWANConfig and ~300 bytes metadata.
Also, an attempt to rejoin initial seeds is performed once a minute. It seems to be the only way to join cluster parts after datacenter connectivity issues long enough to mark another part nodes as failed.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported workload with DefaultWANConfig, about 2,500 nodes, 300-byte metadata, and the once-per-minute seed rejoin behavior. Measure CPU during normal operation and after datacenter connectivity recovery, then determine whether the usage is expected or indicates a configuration or implementation problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.