hashicorp / hashicorp/consul

Consul perpetually loses leadership when provided with an invalid config entry via `config_entries.bootstrap`

Open
#8,228 3 comments 0 reactions 0 assignees View on GitHub
theme/connect
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

If I provide config entries in Consul via `config_entries.bootstrap` property but my config entry is invalid (due to a mistake or something else), Consul will perpetually lose leadership and attempt to transfer it to another server.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a config file to use and save it as `config.json`:
```
{
"config_entries": {
"bootstrap": [
{
"kind": "service-defaults",
"name": "http-service-default",
"protocol": "http"
},
{
"kind": "ingress-gateway",
"name": "ingress-gateway",
"listeners": {
"port": 8080,
"protocol": "http",
"services": [
{
"name": "static-server"
}
]
}
}
]
}
}
```
1. Start consul agent in dev mode using this config file `consul agent -dev -config-dir config.json`
1. See continuous `failed to establish leadership` errors in the logs:

Agent logs

```
==> Starting Consul agent...
Version: 'v1.8.0+ent'
Node ID: 'cedeebca-425d-b2be-f809-f73b313be8c6'
Node name: 'Irynas-MacBook-Pro-2.local'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false

==> Log data will now stream in as it occurs:

2020-07-02T11:15:54.549-0700 [DEBUG] agent: Using random ID as node ID: id=cedeebca-425d-b2be-f809-f73b313be8c6
2020-07-02T11:15:54.549-0700 [WARN] agent: Node name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: node_name=Irynas-MacBook-Pro-2.local
2020-07-02T11:15:54.556-0700 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cedeebca-425d-b2be-f809-f73b313be8c6 Address:127.0.0.1:8300}]"
2020-07-02T11:15:54.557-0700 [INFO] agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader=
2020-07-02T11:15:54.558-0700 [INFO] agent.server.serf.wan: serf: EventMemberJoin: Irynas-MacBook-Pro-2.local.dc1 127.0.0.1
2020-07-02T11:15:54.558-0700 [INFO] agent.server.serf.lan: serf: EventMemberJoin: Irynas-MacBook-Pro-2.local 127.0.0.1
2020-07-02T11:15:54.559-0700 [INFO] agent.server: Handled event for server in area: event=member-join server=Irynas-MacBook-Pro-2.local.dc1 area=wan
2020-07-02T11:15:54.559-0700 [INFO] agent.server: Adding LAN server: server="Irynas-MacBook-Pro-2.local (Addr: tcp/127.0.0.1:8300) (DC: dc1)"
2020-07-02T11:15:54.561-0700 [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=tcp
2020-07-02T11:15:54.561-0700 [INFO] agent: Started DNS server: address=127.0.0.1:8600 network=udp
2020-07-02T11:15:54.563-0700 [INFO] agent: Started HTTP server: address=127.0.0.1:8500 network=tcp
2020-07-02T11:15:54.563-0700 [INFO] agent: Started gRPC server: address=127.0.0.1:8502 network=tcp
2020-07-02T11:15:54.563-0700 [INFO] agent: started state syncer
==> Consul agent running!
2020-07-02T11:15:54.598-0700 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=
2020-07-02T11:15:54.598-0700 [INFO] agent.server.raft: entering candidate state: node="Node at 127.0.0.1:8300 [Candidate]" term=2
2020-07-02T11:15:54.598-0700 [DEBUG] agent.server.raft: votes: needed=1
2020-07-02T11:15:54.598-0700 [DEBUG] agent.server.raft: vote granted: from=cedeebca-425d-b2be-f809-f73b313be8c6 term=2 tally=1
2020-07-02T11:15:54.598-0700 [INFO] agent.server.raft: election won: tally=1
2020-07-02T11:15:54.598-0700 [INFO] agent.server.raft: entering leader state: leader="Node at 127.0.0.1:8300 [Leader]"
2020-07-02T11:15:54.599-0700 [INFO] agent.server: cluster leadership acquired
2020-07-02T11:15:54.599-0700 [INFO] agent.server: New leader elected: payload=Irynas-MacBook-Pro-2.local
2020-07-02T11:15:54.599-0700 [DEBUG] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:8300
2020-07-02T11:15:54.600-0700 [INFO] agent.server: Created the builtin namespace: namespace=default
2020-07-02T11:15:54.600-0700 [INFO] agent.leader: started routine: routine="namespace deferred deletion"
2020-07-02T11:15:54.601-0700 [ERROR] agent.server: failed to establish leadership: error="Failed to apply configuration entry "ingress-gateway" / "ingress-gateway": service "static-server" has protocol "tcp", which does not match defined listener protocol "http""
2020-07-02T11:15:54.601-0700 [DEBUG] agent.leader: stopping routine: routine="namespace deferred deletion"
2020-07-02T11:15:54.601-0700 [DEBUG] agent.leader: stopped routine: routine="namespace deferred deletion"
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:54.601-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:54.601-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:54.601-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:54.601-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:54.601-0700 [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-02T11:15:54.755-0700 [INFO] agent: Synced node info
2020-07-02T11:15:54.755-0700 [DEBUG] agent: Node info in sync
2020-07-02T11:15:56.385-0700 [DEBUG] agent: Node info in sync
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:8300
2020-07-02T11:15:59.602-0700 [INFO] agent.leader: started routine: routine="namespace deferred deletion"
2020-07-02T11:15:59.602-0700 [ERROR] agent.server: failed to establish leadership: error="Failed to apply configuration entry "ingress-gateway" / "ingress-gateway": service "static-server" has protocol "tcp", which does not match defined listener protocol "http""
2020-07-02T11:15:59.602-0700 [DEBUG] agent.leader: stopping routine: routine="namespace deferred deletion"
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:59.602-0700 [DEBUG] agent.leader: stopped routine: routine="namespace deferred deletion"
2020-07-02T11:15:59.602-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:59.602-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:15:59.602-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:15:59.602-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-02T11:15:59.602-0700 [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:8300
2020-07-02T11:16:04.604-0700 [INFO] agent.leader: started routine: routine="namespace deferred deletion"
2020-07-02T11:16:04.604-0700 [ERROR] agent.server: failed to establish leadership: error="Failed to apply configuration entry "ingress-gateway" / "ingress-gateway": service "static-server" has protocol "tcp", which does not match defined listener protocol "http""
2020-07-02T11:16:04.604-0700 [DEBUG] agent.leader: stopping routine: routine="namespace deferred deletion"
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:16:04.604-0700 [DEBUG] agent.leader: stopped routine: routine="namespace deferred deletion"
2020-07-02T11:16:04.604-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="cannot find peer"
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:16:04.604-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="cannot find peer"
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: starting leadership transfer: id= address=
2020-07-02T11:16:04.604-0700 [DEBUG] agent.server.raft: cannot find peer
2020-07-02T11:16:04.604-0700 [ERROR] agent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="cannot find peer"
2020-07-02T11:16:04.604-0700 [ERROR] agent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.