hashicorp / hashicorp/consul

Update Outage doc about recovering multiple servers in a multi-server cluster while keeping servers running

Open
#8,964 1 comment 0 reactions 0 assignees View on GitHub
type/docs
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Feature Description

I had a stupidity based outage this morning. I had a cluster of 3 nodes running fine. I reorganized my terraform, started 3 new nodes for a total of 6. All good. Then I destroyed the old nodes. Back to 3. My cluster died : the leader in the old nodes went away, the remaining couldn't elect a leader as they where trying to reach the old ones to proceed to election.

The Outage doc state that you should stop all your servers as a first step to a manual recovery using peers.json ([around here](https://learn.hashicorp.com/tutorials/consul/recovery-outage?in=consul/day-2-operations#manual-recovery-using-peers-json)).

I couldn't do that, that would have killed service discovery for all my services. But the prospect of leaving everything in a frozen state until this night was giving me cold sweat. I searched for a way to force a leader, of course the "consul operator" didn't help due to no leader, etc. In the end I tried putting a peers.json on one of the node and restart it, the 2 others being still running. And it worked, the election proceeded properly, I got a leader.

I don't know the full implication of what I did. I guess I could have messed up the key/value. But I tried since I've no key/value in Consul except for a couple I can re-create, and I expected Consul to sort it own stuff by itself.

So I'ld suggest someone who understand this fully to add a chapter about it in the outage doc. With all the warning, red flags & so on required off course. That would have helped me greatly.

Just in case

```
Consul v1.8.3
Revision a9322b9c7
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.