Endless election cycle
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Starting a cluster with 3 nodes results in endless election cycle
---
### Consul info for both Client and Server
Server info
Consul.json file
```json
{
"server": true,
"bootstrap_expect": 3,
"data_dir": "/var/consul",
"bind_addr": "10.35.0.8",
"retry_join": [ "10.30.0.13", "10.35.0.8", "10.36.0.12"],
"ui": true
}
```
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = bf0166d8
version = 1.19.0
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr =
server = true
raft:
applied_index = 0
commit_index = 0
fsm_pending = 0
last_contact = never
last_log_index = 181
last_log_term = 210
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9b79f23f-c83e-61ad-f61c-bd45ad20b15f Address:10.36.0.12:8300} {Suffrage:Voter ID:a0965b21-4900-5154-8a71-b5b06c4725e8 Address:10.30.0.13:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Candidate
term = 1439
runtime:
arch = amd64
cpu_count = 4
goroutines = 136
max_procs = 4
os = linux
version = go1.22.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 59
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 38
members = 3
query_queue = 0
query_time = 1
```
```
Server agent HCL config
```
### Operating system and Environment details
Ubuntu 20.04
### Log Fragments
````
Jun 28 14:31:23 resticloud consul[2768931]: 2024-06-28T14:31:23.835+0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1462
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1463
Jun 28 14:31:40 resticloud consul[2768931]: 2024-06-28T14:31:40.469+0500 [ERROR] agent: Coordinate update error: error="No cluster leader"
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1464
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1465
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1466
Jun 28 14:31:57 resticloud consul[2768931]: 2024-06-28T14:31:57.767+0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1467
```
Contributor guide
Research direction
Start with the supplied Consul JSON configuration, server information, and Raft election log fragments; reproduce the three-node setup if possible. Trace why the cluster remains in Candidate state and confirm completion when a leader is elected without repeated election timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100