Ping a node that tries to join the cluster before letting it join

This problem happened in the context of Consul, but the issue is with memberlist inside of Consul.

When a single member of the cluster is unreachable to reach, but able to send messages, it can pollute the gossip cluster with inaccurate and irrelevant information. This can end up happening accidentally pretty easily with forgetting to open a port on an EC2 security group. This problem continues to get worse the more nodes in the cluster that aren't cooperating.

When the appropriate node doesn't refute the message fast enough, it gets kicked from the cluster. For our high availability components this isn't a big problem, albeit annoying, but for our single instance nodes this causes a big problem for DNS.

This gets even worse when this uncooperative node just starts telling the cluster that one of the servers is dead. This can end up causing a leader election if the server doesn't refute the message fast enough.

It would be nice if, on join, memberlist attempted to detect a misbehaving node and kicked it from the cluster before it can cause a cluster outage. This can be as simple as just performing a simple ping and getting a response. Since this would be done on joining the cluster, it's probably better to avoid probeNode (which sends other messages) and only send the ping message.

The issue names the join path, probeNode, and the ping message but no files or tests. Start by tracing how a node is admitted during join and how probeNode sends messages. Done means a joining node is ping-validated before it can affect the gossip cluster, with coverage for an unreachable or misbehaving node.

Written by the indexing model from the issue text.