hashicorp / hashicorp/consul

Certificate rotation failed

Open
#10,919 1 comment 0 reactions 1 assignee View on GitHub

@markan is already working on this.

Since Aug 30, 2021.

theme/certificates type/docs
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I have a 3 node consul cluster setup with local certificates to encrypt the communication. See configuration below. On 08-18-2021 at 4:00 PM the consul follower nodes started to get this error message: `tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2021-08-22T17:58:11-05:00 is after 2021-08-18T21:00:26Z` I did not make any configuration changes on that day. From researching it, it looks like the certificates were not correctly rotated. To get consul working again, I had to set `verify_incoming=false`, `verify_outgoing=false` on the servers/agents and then `auto_encrypt.tls=false` on the agents.
Would like to know how/why this happened but more importantly, how do I force the certificates to rotate again and get the cluster back into a good state?

Config 1:
```
encrypt = "XXXXXXXXX"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/XXX.pem"
key_file = "/etc/consul.d/XXX-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = false
auto_encrypt {
allow_tls = true
}
```

#### Reproduction Steps

I have a consul test cluster setup in the same 3 node configuration and I cannot reproduce the issue

### Consul info for both Client and Server
server:
```
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = 10bb6cb3
version = 1.9.4
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = XXXX:8300
server = true
raft:
applied_index = 30903786
commit_index = 30903786
fsm_pending = 0
last_contact = 0
last_log_index = 30903786
last_log_term = 3382
last_snapshot_index = 30900533
last_snapshot_term = 3382
latest_configuration = [XXXX]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 3382
runtime:
arch = amd64
cpu_count = 1
goroutines = 276
max_procs = 1
os = linux
version = go1.15.8
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1017
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 69481
members = 25
query_queue = 0
query_time = 8
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3832
members = 3
query_queue = 0
query_time = 1
```

### Operating system and Environment details

All nodes running on RHEL 8.4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.