Cluster lost leadership and not able to restart cluster after adding 3 more servers
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
When filing a bug, please include the following:
### `consul version` for both Client and Server
Client: `0.9.2`
Server: `0/9/2`
### `consul info` for both Client and Server
```
agent:
check_monitors = 0
check_ttls = 1
checks = 1
services = 1
build:
prerelease =
revision = 75ca2ca
version = 0.9.2
consul:
bootstrap = false
known_datacenters = 1
leader = false
leader_addr =
server = true
raft:
applied_index = 4441401
commit_index = 4441401
fsm_pending = 0
last_contact = 1.723895932s
last_log_index = 4441403
last_log_term = 3
last_snapshot_index = 4438442
last_snapshot_term = 3
latest_configuration = [{Suffrage:Voter ID:10.30.87.74:8300 Address:10.30.87.74:8300} {Suffrage:Voter ID:10.30.4.29:8300 Address:10.30.4.29:8300} {Suffrage:Voter ID:10.30.134.219:8300 Address:10.30.134.219:8300} {Suffrage:Vot
er ID:10.30.151.60:8300 Address:10.30.151.60:8300} {Suffrage:Voter ID:10.30.72.85:8300 Address:10.30.72.85:8300} {Suffrage:Voter ID:10.30.4.34:8300 Address:10.30.4.34:8300}]
latest_configuration_index = 4441402
num_peers = 5
protocol_version = 2
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 188
runtime:
arch = amd64
cpu_count = 1
goroutines = 91
max_procs = 2
os = linux
version = go1.8.3
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 33
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 94
members = 9
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
centos (amazon ami) on aws.
```
{
"advertise_addr": "10.30.4.29",
"bootstrap": false,
"server": true,
"datacenter": "eu-west-1",
"data_dir": "/var/consul",
"log_level": "INFO",
"bootstrap_expect": 3,
"enable_syslog": true,
"start_join": [],
"retry_join": [
"provider=aws tag_key=Name tag_value=main-consul-vault"
],
"leave_on_terminate": true,
"addresses": {
"http": "0.0.0.0"
},
"ports": {
"http": 8500
}
}
```
### Description of the Issue (and unexpected/desired result)
Had a stable cluster with 3 servers and 3 separate agents.
Added 3 more servers with aws autoscaling to cycle the instances to bigger sizes.
As the new servers joined the cluster, leadership was lost and it doesn't get resolved.
### Reproduction steps
### Log Fragments or Link to [gist](https://gist.github.com/)
https://gist.githubusercontent.com/hamstah/50b39dc72081d7071e72fbdadf9f58c9/raw/b49be50029670a1c6a3e00f2476644d847136fc2/log
```
$ consul members
Node Address Status Type Build Protocol DC
consul-vault-i-012dde7e69f4f2533.test.paywithcurl.tech 10.30.87.74:8301 alive server 0.9.2 2 eu-west-1
consul-vault-i-02d23654286345ba7.test.paywithcurl.tech 10.30.4.34:8301 alive server 0.9.2 2 eu-west-1
consul-vault-i-0349defd933378da1.test.paywithcurl.tech 10.30.134.219:8301 alive server 0.9.2 2 eu-west-1
consul-vault-i-099d87f78c76c5f95.test.paywithcurl.tech 10.30.72.85:8301 alive server 0.9.2 2 eu-west-1
consul-vault-i-0d781a5c3f695c4d7.test.paywithcurl.tech 10.30.4.29:8301 alive server 0.9.2 2 eu-west-1
consul-vault-i-0fd1aa1f84f27a9c5.test.paywithcurl.tech 10.30.151.60:8301 alive server 0.9.2 2 eu-west-1
ecs-cluster-i-00b7681243f17cece.test.paywithcurl.tech 10.30.10.48:8301 alive client 0.9.2 2 eu-west-1
ecs-cluster-i-00e26e5a2905614d0.test.paywithcurl.tech 10.30.138.10:8301 alive client 0.9.2 2 eu-west-1
ecs-cluster-i-0b50e41df7492e851.test.paywithcurl.tech 10.30.82.227:8301 alive client 0.9.2 2 eu-west-1
```
Contributor guide
Assessment
This issue has not been assessed yet.