hashicorp / hashicorp/memberlist

Cannot add new nodes to cluster after one node leaves

Open
#229 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4.1k
Forks
479
Avg merge
15h 58m
Merged PRs (30d)
3

Description

It seems whenever I have a node leave a cluster, then a rejoin occurs, I get `failed acks` and `handler queue full` logs from the node still in the cluster.

Is there any sort of clean-up I need to do to clear the queue or `ack` on rejoin? Everything works fine with joining until one leaves then tries to rejoin...

Scenario:
- Node A begins, first node in cluster.
- Node B joins cluster, connection is fine
- Node B is killed, after 3 failed acks marked as dead
- Node B restarted, the following messages are seen:

Node A:
```
A node has joined: m-127.0.0.1:8888
## NODE B KILLED
2020/11/19 12:14:33 [DEBUG] memberlist: Failed ping: m-127.0.0.1:8888 (timeout reached)
2020/11/19 12:14:34 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:34 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:36 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:37 [INFO] memberlist: Suspect m-127.0.0.1:8888 has failed, no acks received
2020/11/19 12:14:37 [INFO] memberlist: Marking m-127.0.0.1:8888 as failed, suspect timeout reached (0 peer confirmations)
A node has left: m-127.0.0.1:8888
## NODE B RESTARTED
2020/11/19 12:14:39 [DEBUG] memberlist: Stream connection from=127.0.0.1:53980
2020/11/19 12:14:42 [WARN] memberlist: handler queue full, dropping message (3) from=127.0.0.1:8888
2020/11/19 12:14:43 [WARN] memberlist: handler queue full, dropping message (3) from=127.0.0.1:8888
```
_The change in port&name doesn't seem to make a difference. Whether it is the same node with the same name or not_

Node B (on rejoin):
```
2020/11/19 12:14:39 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:4444
2020/11/19 12:14:39 [WARN] memberlist: Refuting a suspect message (from: m-127.0.0.1:8888)
A node has joined: m-127.0.0.1:4444
2020/11/19 12:14:40 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
2020/11/19 12:14:42 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
2020/11/19 12:14:43 [INFO] memberlist: Marking m-127.0.0.1:4444 as failed, suspect timeout reached (0 peer confirmations)
A node has left: m-127.0.0.1:4444
2020/11/19 12:14:43 [INFO] memberlist: Suspect m-127.0.0.1:4444 has failed, no acks received
```

Example Code Snippet:
```go
type eventDelegate struct{}

func (ed *eventDelegate) NotifyJoin(node *memberlist.Node) {
fmt.Println("A node has joined: " + node.String())
}

func (ed *eventDelegate) NotifyLeave(node *memberlist.Node) {
fmt.Println("A node has left: " + node.String())
}

func (ed *eventDelegate) NotifyUpdate(node *memberlist.Node) {
fmt.Println("A node was updated: " + node.String())
}

func BeginClusterDiscovery() {
log.Println("Beginning cluster discovery...")
log.Println(*NodeInterface, *NodePort) // These are taken as cli flags
MemberName = fmt.Sprintf("m-%s:%d", *NodeInterface, *NodePort)
MemberList, _ = memberlist.Create(&memberlist.Config{
ProtocolVersion: 5,
BindAddr: *NodeInterface,
BindPort: *NodePort,
AdvertiseAddr: *NodeInterface,
AdvertisePort: *NodePort,
TCPTimeout: time.Second,
IndirectChecks: 1,
RetransmitMult: 2,
SuspicionMult: 3,
PushPullInterval: 15 * time.Second,
ProbeTimeout: 200 * time.Millisecond,
ProbeInterval: time.Second,
GossipInterval: 100 * time.Millisecond,
GossipToTheDeadTime: 15 * time.Second,
Name: MemberName,
Events: &eventDelegate{},
})
}
```

Contributor guide

Open the contributing guide

Research direction

Start with the example's memberlist.Create configuration and trace the leave, failure detection, and rejoin sequence that produces the failed-ack and handler-queue-full logs. Reproduce the Node A/Node B scenario, then inspect the membership, acknowledgement, and message-handler paths involved. Done means the rejoin no longer leaves the cluster in this state and the scenario is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.