Treat broken Serf connection as a fatal condition
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 24
- Forks
- 16
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 1
Description
[I'm unsure why this happens, but catalyst-node can get into a state where it keeps getting timeouts from 127.0.0.1:7373 when it's trying to connect to Serf](https://eu-metrics-monitoring.livepeer.live/grafana/explore?left=%5B%221667223421698%22,%221667223937697%22,%22Loki%22,%7B%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P8E80F9AEF21F6940%22%7D,%22expr%22:%22%7Bnamespace%3D%5C%22prod%5C%22,%20pod%3D%5C%22prod-catalyst-0%5C%22,%20region%3D%5C%22nyc%5C%22%7D%20%7C~%20%5C%22livepeer-catalyst-node%5C%22%22,%22refId%22:%22A%22%7D%5D&orgId=1). While it might be undesirable to crash _immediately_ in such cases, Serf being broken for any meaningful length of time should be treated as a fatal condition and crash the node.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating catalyst-node's Serf connection handling and reviewing the linked timeout logs for repeated failures to 127.0.0.1:7373. Done means transient connection failures remain tolerated, while a Serf connection that stays broken for a meaningful period causes the node to crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100