hashicorp / hashicorp/consul

Certificate errors do not affect node health

Open
#10,221 3 comments 1 reaction 0 assignees View on GitHub
theme/certificates theme/telemetry type/docs type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I have a cluster of consul servers that are using TLS encryption. All nodes were joined and communicating fine. When I let the certificates expire, the logs are full of errors but the nodes, entries for the "consul" service and the ui all report healthy status. I expected them to show unhealthy.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster with 3 server nodes, gossip encryption and enable tls between them with short-dated certificates.
2. Confirm the cluster is working properly and the nodes are all communicating.
3. Wait until the certificates expire.
4. The ui reports everything is healthy but the logs contain rows of

[ERROR] agent.server.rpc: failed to read byte: conn=from=x.x.x.x:yyyy error="remote error: tls: bad certificate"

### Consul info for both Client and Server

Consul 1.9.4

### Operating system and Environment details

ubuntu 18.04 on aws

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with a three-server cluster using short-lived TLS certificates, then compare the certificate errors in the logs with node, consul service, and UI health reports. Trace how those health statuses are derived and make the expected unhealthy state verifiable after certificate expiry.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.