hashicorp / hashicorp/serf

How to avoid split-brain with three node serf cluster?

Open
#551 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

Hello,
My task is to accomplish the following with serf:
If one node fail, the other shutdown the first node and bring it's IP addresses to the new live node. Now if I shutdown the first node, the second is up (the third node does nothing, it's just for quorum and has not leave.sh event). If I shutdown the second node, the first is bringed up ok. BUT if I turn off the third node both first and two are down/shutdown (because they execute leave.sh). My question is how to avoid this brain-split scenario, so even if the third node is down the leave.sh will not be executed on both first and second server?

This is my serf.conf on both first and second node:

{
"node_name": "serf-testnode",
"bind": "X.X.X.X",
"advertise": "X.X.X.X",
"rejoin_after_leave": true,
"skip_leave_on_interrupt": true,
"leave_on_terminate" : false,
"tags": {
"datacenter": "XXXX"
},
"event_handlers": [
"member-leave=/etc/serf/bin/leave.sh",
"member-failed=/etc/serf/bin/leave.sh",
"member-join=/etc/serf/bin/join.sh"

]
}

The serf.conf on node 3 has not leave.sh (because I want two node cluster only, but cringed up third node to avoid the split brain but it didn't work). In consul I think this is done with bootstrap option but in serf it's not available. Thank you in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.