hashicorp / hashicorp/consul

A way to force or reset consul CA root during leadership failure scenario.

Open
#6,375 3 comments 8 reactions 0 assignees View on GitHub
theme/certificates theme/connect theme/consul-vault type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

While testing and feeling out consul we got it configured with consul connect ca's vault provider and things worked well, however at one point we assumed that we could empty vault and that consul would be able to setup/change the root CA that is baked into the raft data. Once our test consul cluster was in this state, when coming online it would fall in a really fast loop failing to establish leadership with the following error repeated from the server nodes:

```
consul: failed to establish leadership: stored CA root "06:e7:b6:ab:8f:93:c2:50:45:bf:b1:8c:b6:75:74:8f:52:dd:47:85" is not the active root (f1:40:88:39:b7:ef:39:7e:28:ed:4d:f7:89:45:22:5f:75:06:e2:4c)
```

After a lot of hunting through docs and trying different ways to force a leader and get the certificate rolled or switched out we ended up just rebuilding the 3 server nodes to fix this. I think we learned our lesson to never mess around with the vault pki mounts that consul connect ca uses, otherwise the cluster gets into this state and it doesn't seem like you can ever bring it back online. Where it's stuck electing a leader it doesn't seem you can even work with a server node to attempt to fix or roll the CA cert out for a new one. It's actually quite easy to mess this up, all one has to do is mess with the pki mount in vault that consul connect ca is configured to use.

Are there any plans to force, expunge or get rid of the root CA in consul in a scenario like this in order to get things running again and a leader elected? Possibly a way to "re-bootstrap" the consul CA bits?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.