hashicorp / hashicorp/raft

Leadership election stuck in 3 node cluster in Candidate state.

Open
#472 14 comments 1 reaction 0 assignees View on GitHub
waiting-reply
Dominant language
Go
Stars
9.1k
Forks
1.1k
Avg merge
3h 27m
Merged PRs (30d)
2

Description

I had 3 node clusters. One node died suddenly. Expected one of the node to become the leader. But seeing the logs. It's getting a vote from itself and asking from the dead node. But not from the other live server.

```
{"level":"info","ts":"2021-09-06T10:31:34.688Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] duplicate requestVote for same term: term=1222614"}
{"level":"info","ts":"2021-09-06T10:31:35.307Z","caller":"go-hclog/stdlog.go:82","msg":"[WARN] Election timeout reached, restarting election"}
{"level":"info","ts":"2021-09-06T10:31:35.307Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] entering candidate state: node=\"Node at serveraddr [Candidate]\" term=1222615"}
{"level":"info","ts":"2021-09-06T10:31:35.321Z","caller":"go-hclog/stdlog.go:82","msg":"[DEBUG] votes: needed=2"}
{"level":"info","ts":"2021-09-06T10:31:35.321Z","caller":"go-hclog/stdlog.go:82","msg":"[DEBUG] vote granted: from=**Same Server** term=1222615 tally=1"}
{"level":"info","ts":"2021-09-06T10:31:35.322Z","caller":"go-hclog/stdlog.go:82","msg":"[ERROR] failed to make requestVote RPC: target=\"{Voter **Dead server** }\" error=\"dial tcp **dead server**: connect: connection refused\""}
{"level":"info","ts":"2021-09-06T10:31:36.510Z","caller":"go-hclog/stdlog.go:82","msg":"[INFO] duplicate requestVote for same term: term=1222615"}
{"level":"info","ts":"2021-09-06T10:31:37.065Z","caller":"go-hclog/stdlog.go:82","msg":"[WARN] Election timeout reached, restarting election"}
```

The Raft configuration from one of the servers is not listing the other live server.

```
{
"details": {
"applied_index": "151938",
"commit_index": "151938",
"fsm_pending": "0",
"last_contact": "462h40m13.433454323s",
"last_log_index": "152647",
"last_log_term": "29674",
"last_snapshot_index": "147497",
"last_snapshot_term": "29669",
"latest_configuration": "[{Suffrage:Voter ID:ID237:6000 Address:.237:6000} {Suffrage:Voter ID:ID241:6000 Address:.241:6000}]",
"latest_configuration_index": "0",
"num_peers": "1",
"protocol_version": "3",
"protocol_version_max": "3",
"protocol_version_min": "0",
"snapshot_version_max": "1",
"snapshot_version_min": "0",
"state": "Candidate",
"term": "1222401"
}
}
```

Another Server.

```
{
"details": {
"applied_index": "151938",
"commit_index": "151938",
"fsm_pending": "0",
"last_contact": "467h14m2.550357442s",
"last_log_index": "151938",
"last_log_term": "29674",
"last_snapshot_index": "147487",
"last_snapshot_term": "29669",
"latest_configuration": "[{Suffrage:Voter ID:ID236:6000 Address:.236:6000} {Suffrage:Voter ID:ID237:6000 Address:.237:6000} {Suffrage:Voter ID:ID241:6000 Address:.241:6000}]",
"latest_configuration_index": "0",
"num_peers": "2",
"protocol_version": "3",
"protocol_version_max": "3",
"protocol_version_min": "0",
"snapshot_version_max": "1",
"snapshot_version_min": "0",
"state": "Candidate",
"term": "1223003"
}
}
```

Library version used. github.com/hashicorp/raft v1.3.1
Default config is taken from library default config.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported three-node election logs and the differing latest_configuration and num_peers values from the two servers. Reproduce the failure with one node unavailable, then determine why the surviving live server is absent from one configuration. Done means the remaining nodes can elect a leader after a node failure without repeatedly restarting elections.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.