hashicorp / hashicorp/serf

Failed nodes reported as alive

Open
#627 0 comments 0 reactions 1 assignee Claimed by @kyhavlov View on GitHub
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

I'm testing out serf and it seems like a great project. I've hit one issue in my testing so far.

I set up a 3-node serf cluster on 3 VM's. All nodes report alive on all nodes, as expected. Tags update. All seems healthy.

```
ed@agent-two:~$ serf members
agent-two 192.168.2.17:7946 alive
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 alive
```

Then I disconnected the VM's network adaptor on `agent-two`. Agents one and three report agent two is failed as I'd expect:

```
ed@agent-one:~$ serf members
agent-two 192.168.2.17:7946 failed
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 alive
```

```
ed@agent-three:~$ serf members
agent-three 192.168.2.18:7946 alive
agent-two 192.168.2.17:7946 failed
agent-one 192.168.2.6:7946 alive
```

However agent two only reports `agent-one` as having failed where I'd have expected it to report both one and three as failed:

```
ed@agent-two:~$ serf members
agent-two 192.168.2.17:7946 alive
agent-three 192.168.2.18:7946 alive
agent-one 192.168.2.6:7946 failed
```

In the monitor logs on agent two I can see:

```
2021/03/15 16:14:33 [ERR] memberlist: Failed to send ping: write udp 192.168.2.17:7946->192.168.2.18:7946: sendto: network is unreachable
2021/03/15 16:14:34 [ERR] memberlist: Push/Pull with agent-three failed: dial tcp 192.168.2.18:7946: connect: network is unreachable
```

Which suggests to me that `agent-two` knows it can't communicate with `agent-three`, so I'm wondering why it reports agent-three as `alive` rather than `failed`.

I believe this is a bug in the sense that agent-two falsely believes (or reports) it can communicate with at least one other node when in fact it is entirely isolated.

When I reconnect the network adaptor, after a few seconds all nodes report they are all alive again.

FWIW, if I disconnect the network adaptors on agents one and three, and then check agent two, agent two correctly reports one and three are `failed`.

My config is:

```
{
"interface": "ens33"
, "encrypt_key": "7VpgMKMUFTTluPMNHz7YL1gMPDLPPpkETmec1hI/jkc="
, "snapshot_path": "/opt/serf/serf.snapshot"
, "rejoin_after_leave": true
, "profile": "lan"
, "log_level": "warn"
, "tags": {}
}
```

During this time, the snapshot file on agent-two looks like this:

```
alive: agent-two 192.168.2.17:7946
alive: agent-one 192.168.2.6:7946
alive: agent-three 192.168.2.18:7946
clock: 43
not-alive: agent-one
alive: agent-one 192.168.2.6:7946
```

Does anyone have any suggestions?

### Platform details

Ubuntu Focal 20.04.1 LTS VM's on VMWare Fusion 12.1 Pro on macOS Big Sur 11.2.1 using NAT networking.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.