hashicorp / hashicorp/consul

Consul fails to register vault-sealed-check for vault

Open
#5,439 23 comments 5 reactions 0 assignees View on GitHub
needs-investigation theme/consul-vault
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
I am currently running vault server on the same hosts as my consul servers. When upgrading from consul 1.4.0 to consul 1.4.3 vault now fails to register its sealed check with consul 1.4.3 so sealed status is never being reported to consul and im getting the following log messages piling up in my consul and vault logs.

#### Reproduction Steps
1) Install consul server 1.4.3 on a host as server
2) Configure and install Vault server 1.0.2 on same host.
3) Setup consul backend for vault
4) Unseal vault
5) Unseal checks start failing trying to report a check status to consul that does not exist.
6) Rollback to consul 1.4.0 and seal status checks work correctly.

### Operating system and Environment details
Official Docker container for consul 1.4.3 and vault 1.0.2

### Log Fragments

#### Vault Logs
```Vault server configuration:

Api Address: https://192.168.28.63:8200
Cgo: disabled
Cluster Address: https://192.168.28.63:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "192.168.28.63:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: (not set)
Mlock: supported: true, enabled: true
Storage: consul (HA available)
Version: Vault v1.0.2
Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33

==> Vault server started! Log data will stream in below:

2019-03-07T15:26:05.882Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2019-03-07T15:26:08.434Z [INFO] core: vault is unsealed
2019-03-07T15:26:08.434Z [INFO] core: entering standby mode
2019-03-07T15:26:27.479Z [WARN] storage.consul: check unable to talk with Consul backend: error="Unexpected response code: 500 (Unknown check "vault:192.168.28.63:8200:vault-sealed-check")"
```

#### Consul Logs
```
2019/03/07 15:25:58 [INFO] agent: Deregistered service "vault:192.168.28.63:8200"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "mem-util"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "dsk-util"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:05 [INFO] agent: Synced service "vault:192.168.28.63:8200"
2019/03/07 15:26:05 [INFO] agent: Synced check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:08 [INFO] agent: Synced check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:20 [INFO] agent: Synced check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:26:20 [INFO] agent: Synced check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-acfqsqjxbgap2cvwlfedkvyx55twlbsl"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-fwtmi5j5ajxekyuajonqk4pwlauj3dko"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-p4pxc2srsurolx43mc7lztdb7fwnwbh3"
2019/03/07 15:26:22 [ERR] http: Request PUT /v1/agent/check/deregister/a75809917d97ead0eaebb52cfeabe012dc47abc7, error: Unknown check "a75809917d97ead0eaebb52cfeabe012dc47abc7" from=127.0.0.1:51522
2019/03/07 15:26:22 [ERR] http: Request PUT /v1/agent/check/deregister/e408baff1455ac4cab95892718cd7494f61693ff, error: Unknown check "e408baff1455ac4cab95892718cd7494f61693ff" from=127.0.0.1:51522
2019/03/07 15:26:25 [WARN] agent: Check "e408baff1455ac4cab95892718cd7494f61693ff" socket connection failed: dial tcp 0.0.0.0:4648: connect: connection refused
2019/03/07 15:26:27 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:27 [WARN] agent: Check "vault:192.168.28.63:8200:vault-sealed-check" missed TTL, is now critical
2019/03/07 15:26:28 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:29 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:30 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:31 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:32 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:33 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:34 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Consul agent check registration and the /v1/agent/check/pass endpoint shown in the logs, then compare behavior between Consul 1.4.0 and 1.4.3 with Vault 1.0.2. Confirm that vault-sealed-check remains registered after startup and that Vault's pass updates no longer return Unknown check errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.