Improve force-leave
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 609
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 3
Description
Hey,
Thanks for providing Serf. I had an Issue while running multiple instances of Serf
**Setup**:
I have infiniband interface _ib0_ and two Infiniband partitions with IPoIB : lets call them _ib0.x_ and _ib0.y_
I have two serf agents running, each of them binds to one interface and uses the name of that interface i.e. ib0.x and ib0.y as discover for mDNS
in ib0.x _cluster_ there are more nodes than in _ib0.y_ let's call them A, B
It was working just fine until -for some reason i couldn't reproduce- A,B joined joined the cluster ib0.y and they were shown as a live . Checking the logs i could see that serf agent in _ib0.y_ can't send gossip packets to A and B since they are actually not on the same network (one of them is ipv4 and the other is ipv6) yet they were shown as alive
Serf reachability however complains about missing acks from A and B which is correct
**Steps to solve that issue** :
I tried first to stop serf service in A,B the result was in ib0.x there were shown as left but in ib0.y one of them was shown as failed and the other as alive
So A, B were "stuck" in _ib0.y_ cluster and i couldn't remove them even with force-leave the `alive` node was still in leaving state for days
The only work around that worked is by stopping the serf service in all the nodes (200 nodes) in _ib0.y_ and then starting them again
Since the scope of this issue is not to debug that problem since i can't reproduce it But it would be really nice to have a force-leave with a flag that can remove a node or (forget) a node
Sorry for that long issue and thanks a lot in advance
best
Contributor guide
Assessment
This issue has not been assessed yet.