hashicorp / hashicorp/memberlist
Cannot add new nodes to cluster after one node leaves
- Dominant language
- Go
- Stars
- 4.1k
- Forks
- 479
- Avg merge
- 15h 58m
- Merged PRs (30d)
- 3
Description
It seems whenever I have a node leave a cluster, then a rejoin occurs, I get `failed acks` and `handler queue full` logs from the node still in the cluster.
Is there any sort of clean-up I need to do to clear the queue or `ack` on rejoin? Everything works fine with joining until one leaves then tries to rejoin...
Scenario:
- Node A begins, first node in cluster.
- Node B joins cluster, connection is fine
- Node B is killed, after 3 failed acks marked as dead
- Node B restarted, the following messages are seen:
Node A:
```
A node has joined: m-127.0.0.1:8888
## NODE B KILLED
2020/11/19 12:14:33 [DEBUG] memberlist: Failed ping: m-127.0.0.1:8888 (timeout reached)
2020/11/19 12:14:34 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:34 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:36 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:37 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:37 [INFO] memberlist: Marking m-127.0.0.1:8888 as failed, suspect timeout reached (0 peer confirmations)
A node has left: m-127.0.0.1:8888
## NODE B RESTARTED
2020/11/19 12:14:39 [DEBUG] memberlist: Stream connection from=127.0.0.1:53980
2020/11/19 12:14:42 [WARN] memberlist: handler queue full, dropping message (3) from=127.0.0.1:8888
2020/11/19 12:14:43 [WARN] memberlist: handler queue full, dropping message (3) from=127.0.0.1:8888
```
_The change in port&name doesn't seem to make a difference. Whether it is the same node with the same name or not_
Node B (on rejoin):
```
2020/11/19 12:14:39 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:4444
2020/11/19 12:14:39 [WARN] memberlist: Refuting a suspect message (from: m-127.0.0.1:8888)
A node has joined: m-127.0.0.1:4444
2020/11/19 12:14:40 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
2020/11/19 12:14:42 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
2020/11/19 12:14:43 [INFO] memberlist: Marking m-127.0.0.1:4444 as failed, suspect timeout reached (0 peer confirmations)
A node has left: m-127.0.0.1:4444
2020/11/19 12:14:43 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
```
Example Code Snippet:
```go
type eventDelegate struct{}
func (ed *eventDelegate) NotifyJoin(node *memberlist.Node) {
fmt.Println("A node has joined: " + node.String())
}
func (ed *eventDelegate) NotifyLeave(node *memberlist.Node) {
fmt.Println("A node has left: " + node.String())
}
func (ed *eventDelegate) NotifyUpdate(node *memberlist.Node) {
fmt.Println("A node was updated: " + node.String())
}
func BeginClusterDiscovery() {
log.Println("Beginning cluster discovery...")
log.Println(*NodeInterface, *NodePort) // These are taken as cli flags
MemberName = fmt.Sprintf("m-%s:%d", *NodeInterface, *NodePort)
MemberList, _ = memberlist.Create(&memberlist.Config{
ProtocolVersion: 5,
BindAddr: *NodeInterface,
BindPort: *NodePort,
AdvertiseAddr: *NodeInterface,
AdvertisePort: *NodePort,
TCPTimeout: time.Second,
IndirectChecks: 1,
RetransmitMult: 2,
SuspicionMult: 3,
PushPullInterval: 15 * time.Second,
ProbeTimeout: 200 * time.Millisecond,
ProbeInterval: time.Second,
GossipInterval: 100 * time.Millisecond,
GossipToTheDeadTime: 15 * time.Second,
Name: MemberName,
Events: &eventDelegate{},
})
}
```
Contributor guide
Research direction
Start with the example's memberlist.Create configuration and trace the leave, failure detection, and rejoin sequence that produces the failed-ack and handler-queue-full logs. Reproduce the Node A/Node B scenario, then inspect the membership, acknowledgement, and message-handler paths involved. Done means the rejoin no longer leaves the cluster in this state and the scenario is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100