hashicorp / hashicorp/consul

recovering from leader loss: neither force-leave nor cleanup_dead_servers works

Open
#3,151 8 comments 0 reactions 0 assignees View on GitHub
type/docs
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

### `consul version` for both Client and Server
Both Consul v0.8.4

### `consul info` for both Client and Server
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 2
build:
prerelease =
revision = f436077
version = 0.8.4
consul:
bootstrap = false
known_datacenters = 1
leader = false
leader_addr =
server = true
raft:
applied_index = 144
commit_index = 144
fsm_pending = 0
last_contact = 7m32.065037683s
last_log_index = 146
last_log_term = 3
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:f5ac46b4-299d-23f5-6a25-dbb1e0565a58 Address:172.30.50.96:8300} {Suffrage:Voter ID:000449be-6dbb-e34e-993f-a9f8334ea0f9 Address:172.30.31.217:8300} {Suffrage:Nonvoter ID:cb832411-bd03-bd2c-3b2e-40badd7c969e Address:172.30.2.254:8300}]
latest_configuration_index = 145
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Candidate
term = 62
runtime:
arch = amd64
cpu_count = 1
goroutines = 72
max_procs = 1
os = linux
version = go1.8.3
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 3
failed = 0
health_score = 0
intent_queue = 0
left = 3
member_time = 33
members = 7
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 3
health_score = 0
intent_queue = 0
left = 0
member_time = 30
members = 7
query_queue = 0
query_time = 1
```

### Operating system and Environment details
official docker image 37ffadd9b8a6

### Description of the Issue (and unexpected/desired result)

loss of quorum cannot be recovered unless the replacement instances have the same ip.

### Reproduction steps

1. start 5 aws instances with --expect-bootstrap 5
2. wait for consensus.
3. murder any 3 of them and respin 3 different ones with _different ip_
4. raft: Failed to make RequestVote RPC to forever.

force-leave has no effect, neither does CleanupDeadServers = true change anything.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.