hashicorp / hashicorp/consul

Endless election cycle

Open
#21,380 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
Starting a cluster with 3 nodes results in endless election cycle

---

### Consul info for both Client and Server

Server info

Consul.json file
```json
{
"server": true,
"bootstrap_expect": 3,
"data_dir": "/var/consul",
"bind_addr": "10.35.0.8",
"retry_join": [ "10.30.0.13", "10.35.0.8", "10.36.0.12"],
"ui": true
}
```

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = bf0166d8
version = 1.19.0
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr =
server = true
raft:
applied_index = 0
commit_index = 0
fsm_pending = 0
last_contact = never
last_log_index = 181
last_log_term = 210
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9b79f23f-c83e-61ad-f61c-bd45ad20b15f Address:10.36.0.12:8300} {Suffrage:Voter ID:a0965b21-4900-5154-8a71-b5b06c4725e8 Address:10.30.0.13:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Candidate
term = 1439
runtime:
arch = amd64
cpu_count = 4
goroutines = 136
max_procs = 4
os = linux
version = go1.22.4
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 59
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 38
members = 3
query_queue = 0
query_time = 1
```

```
Server agent HCL config
```

### Operating system and Environment details

Ubuntu 20.04

### Log Fragments
````
Jun 28 14:31:23 resticloud consul[2768931]: 2024-06-28T14:31:23.835+0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:25 resticloud consul[2768931]: 2024-06-28T14:31:25.640+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1462
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:34 resticloud consul[2768931]: 2024-06-28T14:31:34.718+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1463
Jun 28 14:31:40 resticloud consul[2768931]: 2024-06-28T14:31:40.469+0500 [ERROR] agent: Coordinate update error: error="No cluster leader"
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:42 resticloud consul[2768931]: 2024-06-28T14:31:42.660+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1464
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:48 resticloud consul[2768931]: 2024-06-28T14:31:48.113+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1465
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:31:55 resticloud consul[2768931]: 2024-06-28T14:31:55.647+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1466
Jun 28 14:31:57 resticloud consul[2768931]: 2024-06-28T14:31:57.767+0500 [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838+0500 [WARN] agent.server.raft: Election timeout reached, restarting election
Jun 28 14:32:03 resticloud consul[2768931]: 2024-06-28T14:32:03.838+0500 [INFO] agent.server.raft: entering candidate state: node="Node at 10.36.0.12:8300 [Candidate]" term=1467
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Consul JSON configuration, server information, and Raft election log fragments; reproduce the three-node setup if possible. Trace why the cluster remains in Candidate state and confirm completion when a leader is elected without repeated election timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.