Failed nodes reported as alive
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 609
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 3
Description
I'm testing out serf and it seems like a great project. I've hit one issue in my testing so far.
I set up a 3-node serf cluster on 3 VM's. All nodes report alive on all nodes, as expected. Tags update. All seems healthy.
```
ed@agent-two:~$ serf members
agent-two 192.168.2.17:7946 alive
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 alive
```
Then I disconnected the VM's network adaptor on `agent-two`. Agents one and three report agent two is failed as I'd expect:
```
ed@agent-one:~$ serf members
agent-two 192.168.2.17:7946 failed
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 alive
```
```
ed@agent-three:~$ serf members
agent-three 192.168.2.18:7946 alive
agent-two 192.168.2.17:7946 failed
agent-one 192.168.2.6:7946 alive
```
However agent two only reports `agent-one` as having failed where I'd have expected it to report both one and three as failed:
```
ed@agent-two:~$ serf members
agent-two 192.168.2.17:7946 alive
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 failed
```
In the monitor logs on agent two I can see:
```
2021/03/15 16:14:33 [ERR] memberlist: Failed to send ping: write udp 192.168.2.17:7946->192.168.2.18:7946: sendto: network is unreachable
2021/03/15 16:14:34 [ERR] memberlist: Push/Pull with agent-three failed: dial tcp 192.168.2.18:7946: connect: network is unreachable
```
Which suggests to me that `agent-two` knows it can't communicate with `agent-three`, so I'm wondering why it reports agent-three as `alive` rather than `failed`.
I believe this is a bug in the sense that agent-two falsely believes (or reports) it can communicate with at least one other node when in fact it is entirely isolated.
When I reconnect the network adaptor, after a few seconds all nodes report they are all alive again.
FWIW, if I disconnect the network adaptors on agents one and three, and then check agent two, agent two correctly reports one and three are `failed`.
My config is:
```
{
"interface": "ens33"
, "encrypt_key": "7VpgMKMUFTTluPMNHz7YL1gMPDLPPpkETmec1hI/jkc="
, "snapshot_path": "/opt/serf/serf.snapshot"
, "rejoin_after_leave": true
, "profile": "lan"
, "log_level": "warn"
, "tags": {}
}
```
During this time, the snapshot file on agent-two looks like this:
```
alive: agent-two 192.168.2.17:7946
alive: agent-one 192.168.2.6:7946
alive: agent-three 192.168.2.18:7946
clock: 43
not-alive: agent-one
alive: agent-one 192.168.2.6:7946
```
Does anyone have any suggestions?
### Platform details
Ubuntu Focal 20.04.1 LTS VM's on VMWare Fusion 12.1 Pro on macOS Big Sur 11.2.1 using NAT networking.
Contributor guide
Assessment
This issue has not been assessed yet.