Leadership election stuck in 3 node cluster in Candidate state.
- Dominant language
- Go
- Stars
- 9.1k
- Forks
- 1.1k
- Avg merge
- 3h 27m
- Merged PRs (30d)
- 2
Description
I had 3 node clusters. One node died suddenly. Expected one of the node to become the leader. But seeing the logs. It's getting a vote from itself and asking from the dead node. But not from the other live server.
```
{"level":"info","ts":"2021-09-06T10:31:34.688Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] duplicate requestVote for same term: term=1222614"}
{"level":"info","ts":"2021-09-06T10:31:35.307Z","caller":"go-hclog/stdlog.go:82","msg":"[WARN] Election timeout reached, restarting election"}
{"level":"info","ts":"2021-09-06T10:31:35.307Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] entering candidate state: node=\"Node at serveraddr [Candidate]\" term=1222615"}
{"level":"info","ts":"2021-09-06T10:31:35.321Z","caller":"go-hclog/stdlog.go:82","msg":"[DEBUG] votes: needed=2"}
{"level":"info","ts":"2021-09-06T10:31:35.321Z","caller":"go-hclog/stdlog.go:82","msg":"[DEBUG] vote granted: from=**Same Server** term=1222615 tally=1"}
{"level":"info","ts":"2021-09-06T10:31:35.322Z","caller":"go-hclog/stdlog.go:82","msg":"[ERROR] failed to make requestVote RPC: target=\"{Voter **Dead server** }\" error=\"dial tcp **dead server**: connect: connection refused\""}
{"level":"info","ts":"2021-09-06T10:31:36.510Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] duplicate requestVote for same term: term=1222615"}
{"level":"info","ts":"2021-09-06T10:31:37.065Z","caller":"go-hclog/stdlog.go:82","msg":"[WARN] Election timeout reached, restarting election"}
```
The Raft configuration from one of the servers is not listing the other live server.
```
{
"details": {
"applied_index": "151938",
"commit_index": "151938",
"fsm_pending": "0",
"last_contact": "462h40m13.433454323s",
"last_log_index": "152647",
"last_log_term": "29674",
"last_snapshot_index": "147497",
"last_snapshot_term": "29669",
"latest_configuration": "[{Suffrage:Voter ID:ID237:6000 Address:.237:6000} {Suffrage:Voter ID:ID241:6000 Address:.241:6000}]",
"latest_configuration_index": "0",
"num_peers": "1",
"protocol_version": "3",
"protocol_version_max": "3",
"protocol_version_min": "0",
"snapshot_version_max": "1",
"snapshot_version_min": "0",
"state": "Candidate",
"term": "1222401"
}
}
```
Another Server.
```
{
"details": {
"applied_index": "151938",
"commit_index": "151938",
"fsm_pending": "0",
"last_contact": "467h14m2.550357442s",
"last_log_index": "151938",
"last_log_term": "29674",
"last_snapshot_index": "147487",
"last_snapshot_term": "29669",
"latest_configuration": "[{Suffrage:Voter ID:ID236:6000 Address:.236:6000} {Suffrage:Voter ID:ID237:6000 Address:.237:6000} {Suffrage:Voter ID:ID241:6000 Address:.241:6000}]",
"latest_configuration_index": "0",
"num_peers": "2",
"protocol_version": "3",
"protocol_version_max": "3",
"protocol_version_min": "0",
"snapshot_version_max": "1",
"snapshot_version_min": "0",
"state": "Candidate",
"term": "1223003"
}
}
```
Library version used. github.com/hashicorp/raft v1.3.1
Default config is taken from library default config.
Contributor guide
Research direction
Start by reviewing the reported three-node election logs and the differing latest_configuration and num_peers values from the two servers. Reproduce the failure with one node unavailable, then determine why the surviving live server is absent from one configuration. Done means the remaining nodes can elect a leader after a node failure without repeatedly restarting elections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100