consul server become to nonvoter and fails to select a leader
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
we have three servers node1, node2, node3
version is
[cloud@node2 ~]$ consul version
Consul v1.9.2
Revision 6530cf370
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
config file looks like , the difference is the ip addr and name.
{
"addresses": {
"http": "10.10.10.11"
},
"autopilot": {
"cleanup_dead_servers": false
},
"bind_addr": "10.10.10.11",
"bootstrap_expect": 3,
"data_dir": "/tmp/consul_storage",
"datacenter": "storage",
"enable_script_checks": true,
"log_level": "INFO",
"node_name": "node1",
"ports": {
"dns": -1
},
"reconnect_timeout": "8760h",
"retry_join": [
"10.10.10.14",
"10.10.10.17"
],
"server": true,
}
when boot, it fails to choose a leader, this is the first time i see .
using consul monitor --log-level=debug -http-addr=10.10.10.17:8500
log continue looks like
2023-12-22T15:49:38.878+0800 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2023-12-22T15:49:38.878+0800 [INFO] agent.server.raft: entering candidate state: node="Node at 10.10.10.17:8300 [Candidate]" term=47991
2023-12-22T15:49:38.879+0800 [DEBUG] agent.server.raft: votes: needed=2
2023-12-22T15:49:38.879+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=19a68ea7-ac67-a776-3c0f-c6197cbfba7f fallback=10.10.10.17:8300 error="Could not find address for server id 19a68ea7-ac67-a776-3c0f-c6197cbfba7f"
2023-12-22T15:49:38.879+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=54c79a65-1d3c-ad25-ed60-0e200bf0d283 fallback=10.10.10.14:8300 error="Could not find address for server id 54c79a65-1d3c-ad25-ed60-0e200bf0d283"
2023-12-22T15:49:38.881+0800 [DEBUG] agent.server.raft: vote granted: from=19a68ea7-ac67-a776-3c0f-c6197cbfba7f term=47991 tally=1
2023-12-22T15:49:38.882+0800 [DEBUG] agent.server.raft: vote granted: from=54c79a65-1d3c-ad25-ed60-0e200bf0d283 term=47991 tally=2
2023-12-22T15:49:38.882+0800 [INFO] agent.server.raft: election won: tally=2
2023-12-22T15:49:38.882+0800 [INFO] agent.server.raft: entering leader state: leader="Node at 10.10.10.17:8300 [Leader]"
2023-12-22T15:49:38.882+0800 [INFO] agent.server.raft: added peer, starting replication: peer=19a68ea7-ac67-a776-3c0f-c6197cbfba7f
2023-12-22T15:49:38.882+0800 [INFO] agent.server.raft: added peer, starting replication: peer=54c79a65-1d3c-ad25-ed60-0e200bf0d283
2023-12-22T15:49:38.882+0800 [INFO] agent.server.raft: added peer, starting replication: peer=e4d66e88-1359-fa07-672b-a75211472427
2023-12-22T15:49:38.882+0800 [INFO] agent.server: New leader elected: payload=node2
2023-12-22T15:49:38.882+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=19a68ea7-ac67-a776-3c0f-c6197cbfba7f fallback=10.10.10.17:8300 error="Could not find address for server id 19a68ea7-ac67-a776-3c0f-c6197cbfba7f"
2023-12-22T15:49:38.882+0800 [INFO] agent.server: cluster leadership acquired
2023-12-22T15:49:38.882+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=54c79a65-1d3c-ad25-ed60-0e200bf0d283 fallback=10.10.10.14:8300 error="Could not find address for server id 54c79a65-1d3c-ad25-ed60-0e200bf0d283"
2023-12-22T15:49:38.883+0800 [WARN] agent.server.raft: unable to get address for server, using fallback address: id=54c79a65-1d3c-ad25-ed60-0e200bf0d283 fallback=10.10.10.14:8300 error="Could not find address for server id 54c79a65-1d3c-ad25-ed60-0e200bf0d283"
2023-12-22T15:49:38.883+0800 [INFO] agent.server.raft: pipelining replication: peer="{Voter 54c79a65-1d3c-ad25-ed60-0e200bf0d283 10.10.10.14:8300}"
2023-12-22T15:49:38.884+0800 [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter e4d66e88-1359-fa07-672b-a75211472427 10.10.10.11:8300}"
2023-12-22T15:49:38.885+0800 [INFO] agent.server.raft: entering follower state: follower="Node at 10.10.10.17:8300 [Follower]" leader=
2023-12-22T15:49:38.885+0800 [DEBUG] agent.server: shutting down leader loop
2023-12-22T15:49:38.885+0800 [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 54c79a65-1d3c-ad25-ed60-0e200bf0d283 10.10.10.14:8300}"
2023-12-22T15:49:38.885+0800 [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter e4d66e88-1359-fa07-672b-a75211472427 10.10.10.11:8300}"
2023-12-22T15:49:38.885+0800 [ERROR] agent.server: failed to wait for barrier: error="leadership lost while committing log"
2023-12-22T15:49:38.885+0800 [INFO] agent.server: cluster leadership lost
2023-12-22T15:49:39.039+0800 [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2023-12-22T15:49:39.092+0800 [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2023-12-22T15:49:39.103+0800 [DEBUG] agent.server.serf.lan: serf: messageUserEventType: consul:new-leader
2023-12-22T15:49:39.677+0800 [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=10.10.10.17:32892 latency=38.52µs
2023-12-22T15:49:40.106+0800 [WARN] agent: Syncing node info failed.: error="No cluster leader"
My question is why 10.10.10.11 become to Nonvoter , Does this relate to the reason of not select a leader
Contributor guide
Assessment
This issue has not been assessed yet.