hashicorp / hashicorp/vault-helm
Vault with mysql backend goes into sealed status
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 898
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug**
Vault goes into sealed status while trying to release HA lock .
**To Reproduce**
Steps to reproduce the behavior:
1. Install chart with mysql backend (chart version 0.13.0), vault version 1.7.3
2. Vault runs normally with everything up
3. Once in a while few of the pods , in our case 2 out of 3 pods go into sealed status while releasing HA lock.
**Found the following error message :**
"Readiness probe failed: Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed true Total Recovery Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 1.7.3 Storage Type mysql HA Enabled true"
**Vault logs:**
`2021-08-12T18:37:04.287Z [INFO] core: successfully setup plugin catalog: plugin-directory=
2021-08-12T18:37:04.293Z [INFO] core: successfully mounted backend: type=system path=sys/
2021-08-12T18:37:04.294Z [INFO] core: successfully mounted backend: type=identity path=identity/
2021-08-12T18:37:04.296Z [INFO] core: successfully mounted backend: type=kv path=secret/
2021-08-12T18:37:04.297Z [INFO] core: successfully mounted backend: type=ssh path=ssh-client/
2021-08-12T18:37:04.297Z [INFO] core: successfully mounted backend: type=database path=database/
2021-08-12T18:37:04.297Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2021-08-12T18:37:04.297Z [INFO] secrets.database.database_fbea74c1: initializing database rotation queue
2021-08-12T18:37:04.327Z [INFO] core: successfully enabled credential backend: type=token path=token/
2021-08-12T18:37:04.327Z [INFO] core: successfully enabled credential backend: type=kubernetes path=kubernetes/
2021-08-12T18:37:04.328Z [INFO] core: successfully enabled credential backend: type=github path=github/
2021-08-12T18:37:04.328Z [INFO] core: successfully enabled credential backend: type=userpass path=userpass/
2021-08-12T18:37:04.337Z [INFO] rollback: starting rollback manager
2021-08-12T18:37:04.338Z [INFO] core: restoring leases
2021-08-12T18:37:04.483Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
[mysql] 2021/08/12 18:37:04 packets.go:36: read tcp 10.244.7.4:53942->10.161.20.11:3306: read: connection reset by peer
2021-08-12T18:37:04.566Z [ERROR] expiration: error restoring leases: error="failed to read lease entry auth/kubernetes/login/h1390eabaf7f6eb15878f67709849ad7bbd9f5cb5abfa539a7ace940fc3e0a886: invalid connection"
2021-08-12T18:37:04.566Z [ERROR] core: shutting down
2021-08-12T18:37:04.566Z [INFO] core: marked as sealed
[mysql] 2021/08/12 18:37:04 packets.go:36: read tcp 10.244.7.4:53938->10.161.20.11:3306: read: connection reset by peer
[mysql] 2021/08/12 18:37:04 packets.go:36: read tcp 10.244.7.4:53934->10.161.20.11:3306: read: connection reset by peer
[mysql] 2021/08/12 18:37:04 packets.go:36: read tcp 10.244.7.4:53840->10.161.20.11:3306: read: connection reset by peer
2021-08-12T18:37:04.647Z [INFO] identity: entities restored
2021-08-12T18:37:04.653Z [INFO] identity: groups restored
2021-08-12T18:37:05.063Z [INFO] core: usage gauge collection is disabled
2021-08-12T18:37:05.078Z [INFO] core: post-unseal setup complete
2021-08-12T18:37:05.079Z [INFO] core: pre-seal teardown starting
2021-08-12T18:37:05.100Z [INFO] secrets.database.database_fbea74c1: populating role rotation queue
2021-08-12T18:37:05.122Z [INFO] secrets.database.database_fbea74c1: starting periodic ticker
2021-08-12T18:37:05.490Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-08-12T18:37:05.579Z [INFO] rollback: stopping rollback manager
2021-08-12T18:37:05.579Z [INFO] core: pre-seal teardown complete
2021-08-12T18:37:05.579Z [INFO] secrets.database.database_fbea74c1: stopping periodic ticker
2021-08-12T18:37:06.814Z [INFO] http: TLS handshake error from 10.244.10.138:55634: EOF
2021-08-12T18:37:06.927Z [INFO] http: TLS handshake error from 10.244.10.138:55638: EOF
2021-08-12T18:37:07.319Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-08-12T18:37:09.614Z [INFO] http: TLS handshake error from 10.244.10.138:55700: EOF
2021-08-12T18:37:10.003Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-08-12T18:37:10.320Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-08-12T18:37:14.384Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2021-08-12T18:37:15.623Z [INFO] core: stopping cluster listeners
2021-08-12T18:37:15.623Z [INFO] core.cluster-listener: forwarding rpc listeners stopped
2021-08-12T18:37:15.884Z [INFO] core.cluster-listener: rpc listeners successfully shut down
2021-08-12T18:37:15.884Z [INFO] core: cluster listeners successfully shut down
2021-08-12T18:37:25.914Z [INFO] core: vault is sealed`
Other useful info to include: vault pod logs, `kubectl describe statefulset vault` and `kubectl get statefulset vault -o yaml` output
**Expected behavior**
Vault remains operational
**Environment**
* Kubernetes version: v1.19.9
* Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): aks
* Other configuration options or runtime services (istio, etc.): na
* vault-helm version: 1.7.3
Chart values:
```yaml
vault:
server:
ingress:
hosts:
- host: "vault.************************”
paths:
- /
tls:
- hosts:
- vault.*******************
secretName: external-ingress-certificate
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: “*”***********************************”
image:
tag: "1.7.3"
logLevel: "debug"
logFormat: "standard"
resources:
requests:
cpu: 200m
memory: 200Mi
limits:
cpu: 500m
memory: 1.5Gi
injector:
image:
tag: "0.10.2"
agentImage:
tag: "1.7.3"
resources:
requests:
cpu: 20m
memory: 20Mi
limits:
cpu: 100m
memory: 50Mi
```
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.