Update Outage doc about recovering multiple servers in a multi-server cluster while keeping servers running
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Feature Description
I had a stupidity based outage this morning. I had a cluster of 3 nodes running fine. I reorganized my terraform, started 3 new nodes for a total of 6. All good. Then I destroyed the old nodes. Back to 3. My cluster died : the leader in the old nodes went away, the remaining couldn't elect a leader as they where trying to reach the old ones to proceed to election.
The Outage doc state that you should stop all your servers as a first step to a manual recovery using peers.json ([around here](https://learn.hashicorp.com/tutorials/consul/recovery-outage?in=consul/day-2-operations#manual-recovery-using-peers-json)).
I couldn't do that, that would have killed service discovery for all my services. But the prospect of leaving everything in a frozen state until this night was giving me cold sweat. I searched for a way to force a leader, of course the "consul operator" didn't help due to no leader, etc. In the end I tried putting a peers.json on one of the node and restart it, the 2 others being still running. And it worked, the election proceeded properly, I got a leader.
I don't know the full implication of what I did. I guess I could have messed up the key/value. But I tried since I've no key/value in Consul except for a couple I can re-create, and I expected Consul to sort it own stuff by itself.
So I'ld suggest someone who understand this fully to add a chapter about it in the outage doc. With all the warning, red flags & so on required off course. That would have helped me greatly.
Just in case
```
Consul v1.8.3
Revision a9322b9c7
```
Contributor guide
Assessment
This issue has not been assessed yet.