Consul doesn't perform Health Checks regularly for an external service
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
As an extra security measure (beyond the best practice), we've decided that Vault would reside on a separated Consul cluster (opposed to deploying on the same cluster between separating on different Consul DCs). We have configured access to vault through our production (non-vault) cluster as an External Service, using HTTP health checks to the vault cluster.
Unfortunately, it doesn't work as expected, as we're experiencing delayed and inconsistent health check information on our servers.
To emphasize: currently, when we use the API health check for Vault manually, we see that all Vault nodes are working as expected, although in Consul, the health checks claim that one of our servers is down. When the raft leader changes, the supposed timeout error is moving to another Vault node. Both results are relevant for health checks done months ago.
As we are pretty stuck in the current situation now and finding no information about it, we would like to know if the issue is fixable, considered as a bug and still relevant or has been fixed already.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Create a cluster with 7 server nodes on 3 different dcs, operating under the same Consul dc. One of the DCs is operating as a witness, with opened ports only to the Consul Servers.
2. Register Vault nodes as external nodes, with HTTP GET Health Check in HTTPS for /v1/sys/health.
3. Configure interval of 5 seconds, timeout of 1 second.
4. Keep it working until the nodes show inconsistencies.
### Consul info
Server info
```
build:
version = 1.6.1
```
### Operating system
RHEL 7.4, Linux 3.10.0-693.11.6.e17.x86_64
### Log Fragments
Unfortunately we didn't find any relevant log segment for the situation here.
Contributor guide
Assessment
This issue has not been assessed yet.