Certificate rotation failed
@markan is already working on this.
Since Aug 30, 2021.
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I have a 3 node consul cluster setup with local certificates to encrypt the communication. See configuration below. On 08-18-2021 at 4:00 PM the consul follower nodes started to get this error message: `tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2021-08-22T17:58:11-05:00 is after 2021-08-18T21:00:26Z` I did not make any configuration changes on that day. From researching it, it looks like the certificates were not correctly rotated. To get consul working again, I had to set `verify_incoming=false`, `verify_outgoing=false` on the servers/agents and then `auto_encrypt.tls=false` on the agents.
Would like to know how/why this happened but more importantly, how do I force the certificates to rotate again and get the cluster back into a good state?
Config 1:
```
encrypt = "XXXXXXXXX"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/XXX.pem"
key_file = "/etc/consul.d/XXX-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = false
auto_encrypt {
allow_tls = true
}
```
#### Reproduction Steps
I have a consul test cluster setup in the same 3 node configuration and I cannot reproduce the issue
### Consul info for both Client and Server
server:
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 10bb6cb3
version = 1.9.4
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = XXXX:8300
server = true
raft:
applied_index = 30903786
commit_index = 30903786
fsm_pending = 0
last_contact = 0
last_log_index = 30903786
last_log_term = 3382
last_snapshot_index = 30900533
last_snapshot_term = 3382
latest_configuration = [XXXX]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 3382
runtime:
arch = amd64
cpu_count = 1
goroutines = 276
max_procs = 1
os = linux
version = go1.15.8
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1017
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 69481
members = 25
query_queue = 0
query_time = 8
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3832
members = 3
query_queue = 0
query_time = 1
```
### Operating system and Environment details
All nodes running on RHEL 8.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.