hashicorp / hashicorp/serf

Improving (hopefully ;) Serf reconnection ability

Open
#333 7 comments 3 reactions 0 assignees View on GitHub
enhancement thinking
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

Consul agent will try to rejoin only "last seen alive" nodes on restart, which can effectively isolate nodes from the cluster (e.g. see hashicorp/consul#1271) and render cluster inoperable.

I believe this is a limitation of Serf, as Snapshotter contains only alive nodes.

There are several ways to solve this. I can fix it, but I'd like to agree with you on fix design.
For now I'm inclined to:
1. change serf to distinguish "failed" nodes and the nodes that "left" (add failedNodes to the Snapshotter; change Snapshotter.processMemberEvent; change Snapshotter.replay)
2. change serf.Create to rejoin not only previously alive nodes but also nodes that were previously failed

Any comments?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.