Improving (hopefully ;) Serf reconnection ability
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 609
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 3
Description
Consul agent will try to rejoin only "last seen alive" nodes on restart, which can effectively isolate nodes from the cluster (e.g. see hashicorp/consul#1271) and render cluster inoperable.
I believe this is a limitation of Serf, as Snapshotter contains only alive nodes.
There are several ways to solve this. I can fix it, but I'd like to agree with you on fix design.
For now I'm inclined to:
1. change serf to distinguish "failed" nodes and the nodes that "left" (add failedNodes to the Snapshotter; change Snapshotter.processMemberEvent; change Snapshotter.replay)
2. change serf.Create to rejoin not only previously alive nodes but also nodes that were previously failed
Any comments?
Contributor guide
Assessment
This issue has not been assessed yet.