hashicorp / hashicorp/serf

When the nodes leave and start at the same time in a large-capacity consul cluster, some member states will continue to keep leaving, not alive.

Open
#662 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

I am in a consul scenario with 3 servers and 150 clients.
When 150 clients leave at the same time and restart, some member states show leaving
Further analysis, the following scenarios appear
- Node A and Node B are both rebooted
- After node A starts, it receives the leave message from B, and A caches the type and ltime of the message to recentIntents
- When A receives the join message from B, the join message is smaller than the leave message ltime. As a result, In A consul members, the state of B is always leaving.

The source code is as follows

https://github.com/hashicorp/serf/blob/830be12c38e509faa64a2a650629e91c8930a040/serf/serf.go#L1111

https://github.com/hashicorp/serf/blob/830be12c38e509faa64a2a650629e91c8930a040/serf/serf.go#L1220

https://github.com/hashicorp/serf/blob/830be12c38e509faa64a2a650629e91c8930a040/serf/serf.go#L956

https://github.com/hashicorp/serf/blob/830be12c38e509faa64a2a650629e91c8930a040/serf/serf.go#L1225

/kind bug

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.