Serf agent spins CPU if some members does have support multicast
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 609
- Avg merge
- 15h 1m
- Merged PRs (30d)
- 3
Description
I am not sure how Serf goes into this state exactly, but the symptom:
When some of the serf members does not support IP multicast (kernel or ip route), the serf agent will keep spinning CPU cycles trying reconnect to "unreachable" members.
```
2016/01/15 00:20:44 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54573->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:44 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:47 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54574->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:47 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:49 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54575->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:49 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:51 [ERR] memberlist: Failed TCP fallback ping: read tcp 192.168.63.125:54577->192.168.63.111:7946: i/o timeout
2016/01/15 00:20:51 [INFO] memberlist: Suspect node1 has failed, no acks received
2016/01/15 00:20:52 [INFO] memberlist: Marking node1 as failed, suspect timeout reached
```
As you can see from the above, the agent will keep trying to reconnect to the node. Perhaps some sort of exponential back up is necessary to avoid hammering the CPU & I/O subsystem?
For reference: On a BeagleBone Black class device, up to 25% of the CPU is consumed. May not be too noticeable on desktop/server class machine.
Discussion:
https://groups.google.com/d/msg/serfdom/3mipto6NvmI/xQJPm2HVEAAJ
Contributor guide
Assessment
This issue has not been assessed yet.