Consul fails to register vault-sealed-check for vault
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I am currently running vault server on the same hosts as my consul servers. When upgrading from consul 1.4.0 to consul 1.4.3 vault now fails to register its sealed check with consul 1.4.3 so sealed status is never being reported to consul and im getting the following log messages piling up in my consul and vault logs.
#### Reproduction Steps
1) Install consul server 1.4.3 on a host as server
2) Configure and install Vault server 1.0.2 on same host.
3) Setup consul backend for vault
4) Unseal vault
5) Unseal checks start failing trying to report a check status to consul that does not exist.
6) Rollback to consul 1.4.0 and seal status checks work correctly.
### Operating system and Environment details
Official Docker container for consul 1.4.3 and vault 1.0.2
### Log Fragments
#### Vault Logs
```Vault server configuration:
Api Address: https://192.168.28.63:8200
Cgo: disabled
Cluster Address: https://192.168.28.63:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "192.168.28.63:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: (not set)
Mlock: supported: true, enabled: true
Storage: consul (HA available)
Version: Vault v1.0.2
Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33
==> Vault server started! Log data will stream in below:
2019-03-07T15:26:05.882Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2019-03-07T15:26:08.434Z [INFO] core: vault is unsealed
2019-03-07T15:26:08.434Z [INFO] core: entering standby mode
2019-03-07T15:26:27.479Z [WARN] storage.consul: check unable to talk with Consul backend: error="Unexpected response code: 500 (Unknown check "vault:192.168.28.63:8200:vault-sealed-check")"
```
#### Consul Logs
```
2019/03/07 15:25:58 [INFO] agent: Deregistered service "vault:192.168.28.63:8200"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "mem-util"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "dsk-util"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:25:58 [INFO] agent: Deregistered check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:05 [INFO] agent: Synced service "vault:192.168.28.63:8200"
2019/03/07 15:26:05 [INFO] agent: Synced check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:08 [INFO] agent: Synced check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:20 [INFO] agent: Synced check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:26:20 [INFO] agent: Synced check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-acfqsqjxbgap2cvwlfedkvyx55twlbsl"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "vault:192.168.28.63:8200:vault-sealed-check"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "e408baff1455ac4cab95892718cd7494f61693ff"
2019/03/07 15:26:22 [INFO] agent: Deregistered check "a75809917d97ead0eaebb52cfeabe012dc47abc7"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-fwtmi5j5ajxekyuajonqk4pwlauj3dko"
2019/03/07 15:26:22 [INFO] agent: Deregistered service "_nomad-server-p4pxc2srsurolx43mc7lztdb7fwnwbh3"
2019/03/07 15:26:22 [ERR] http: Request PUT /v1/agent/check/deregister/a75809917d97ead0eaebb52cfeabe012dc47abc7, error: Unknown check "a75809917d97ead0eaebb52cfeabe012dc47abc7" from=127.0.0.1:51522
2019/03/07 15:26:22 [ERR] http: Request PUT /v1/agent/check/deregister/e408baff1455ac4cab95892718cd7494f61693ff, error: Unknown check "e408baff1455ac4cab95892718cd7494f61693ff" from=127.0.0.1:51522
2019/03/07 15:26:25 [WARN] agent: Check "e408baff1455ac4cab95892718cd7494f61693ff" socket connection failed: dial tcp 0.0.0.0:4648: connect: connection refused
2019/03/07 15:26:27 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:27 [WARN] agent: Check "vault:192.168.28.63:8200:vault-sealed-check" missed TTL, is now critical
2019/03/07 15:26:28 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:29 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:30 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:31 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:32 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:33 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
2019/03/07 15:26:34 [ERR] http: Request PUT /v1/agent/check/pass/vault:192.168.28.63:8200:vault-sealed-check?note=Vault+Unsealed, error: Unknown check "vault:192.168.28.63:8200:vault-sealed-check" from=127.0.0.1:51576
```
Contributor guide
Research direction
Start by tracing the Consul agent check registration and the /v1/agent/check/pass endpoint shown in the logs, then compare behavior between Consul 1.4.0 and 1.4.3 with Vault 1.0.2. Confirm that vault-sealed-check remains registered after startup and that Vault's pass updates no longer return Unknown check errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100