How to avoid split-brain with three node serf cluster?
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 609
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 3
Description
Hello,
My task is to accomplish the following with serf:
If one node fail, the other shutdown the first node and bring it's IP addresses to the new live node. Now if I shutdown the first node, the second is up (the third node does nothing, it's just for quorum and has not leave.sh event). If I shutdown the second node, the first is bringed up ok. BUT if I turn off the third node both first and two are down/shutdown (because they execute leave.sh). My question is how to avoid this brain-split scenario, so even if the third node is down the leave.sh will not be executed on both first and second server?
This is my serf.conf on both first and second node:
{
"node_name": "serf-testnode",
"bind": "X.X.X.X",
"advertise": "X.X.X.X",
"rejoin_after_leave": true,
"skip_leave_on_interrupt": true,
"leave_on_terminate" : false,
"tags": {
"datacenter": "XXXX"
},
"event_handlers": [
"member-leave=/etc/serf/bin/leave.sh",
"member-failed=/etc/serf/bin/leave.sh",
"member-join=/etc/serf/bin/join.sh"
]
}
The serf.conf on node 3 has not leave.sh (because I want two node cluster only, but cringed up third node to avoid the split brain but it didn't work). In consul I think this is done with bootstrap option but in serf it's not available. Thank you in advance!
Contributor guide
Assessment
This issue has not been assessed yet.