hashicorp / hashicorp/consul

Consul works as a Vault backend, but with many false unhealthy members

Open
#6,876 3 comments 4 reactions 0 assignees View on GitHub
theme/consul-vault
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Even though Consul perfectly works as a Vault backend I am getting these errors in logs all the time:

```
devops-consul-devops-consul-server-2 consul 2019/12/04 08:36:44 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul 2019/12/04 08:36:45 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul 2019/12/04 08:36:46 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul 2019/12/04 08:36:47 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
devops-consul-devops-consul-server-2 consul 2019/12/04 08:36:48 [ERR] http: Request PUT /v1/agent/check/pass/vault:100.118.0.16:8200:vault-sealed-check?note=Vault+Unsealed, error: CheckID "vault:100.118.0.16:8200:vault-sealed-check" does not have associated TTL from=100.118.0.16:33514
```

### Consul info for both Client and Server

Server info

```
agent:
check_monitors = 0
check_ttls = 4
checks = 4
services = 4
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 100.102.200.18:8300
server = true
raft:
applied_index = 129450973
commit_index = 129450973
fsm_pending = 0
last_contact = 0
last_log_index = 129450973
last_log_term = 45
last_snapshot_index = 129440101
last_snapshot_term = 45
latest_configuration = [{Suffrage:Voter ID:9c4f8adc-e40c-eed2-c291-bab53b9f94b4 Address:100.102.200.18:8300} {Suffrage:Voter ID:e6d7e185-c36f-cb45-cc1c-430e20b6cd71 Address:100.102.16.17:8300} {Suffrage:Voter ID:af37afcd-bdaa-8e52-fe2f-94b333e08b7f Address:100.104.176.16:8300}]
latest_configuration_index = 128342841
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 45
runtime:
arch = amd64
cpu_count = 16
goroutines = 243
max_procs = 16
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 22
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 12
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 12
members = 3
query_queue = 0
query_time = 1
```

(For some reason on Consul UI, we have some duplicates and it randomly turns on and off) - **_but there are always at least 3 healthy checks according to the UI._**
![image](https://user-images.githubusercontent.com/6129986/70128805-d8aa8b80-167d-11ea-93be-cabe0085bf5e.png)

### Environment details
Consul version: `1.6.2`
Vault version: `1.3.0`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.