hashicorp / hashicorp/serf

Serf agent spins CPU if some members does have support multicast

Open
#372 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
6.1k
Forks
609
Avg merge
15h 1m
Merged PRs (30d)
3

Description

I am not sure how Serf goes into this state exactly, but the symptom:
When some of the serf members does not support IP multicast (kernel or ip route), the serf agent will keep spinning CPU cycles trying reconnect to "unreachable" members.

```
2016/01/15 00:20:44 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54573->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:44 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:47 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54574->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:47 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:49 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54575->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:49 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:51 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54577->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:51 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:52 [INFO] memberlist: Marking node1 as failed, suspect timeout reached
```

As you can see from the above, the agent will keep trying to reconnect to the node. Perhaps some sort of exponential back up is necessary to avoid hammering the CPU & I/O subsystem?

For reference: On a BeagleBone Black class device, up to 25% of the CPU is consumed. May not be too noticeable on desktop/server class machine.

Discussion:
https://groups.google.com/d/msg/serfdom/3mipto6NvmI/xQJPm2HVEAAJ

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.