hashicorp / hashicorp/vault-helm
Vault snapshot cause cluster failure
- Dominant language
- Shell
- Stars
- 1.3k
- Forks
- 898
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 1
Description
**Describe the bug**
I cannot reproduce this bug because we had this issue on production EKS cluster, but it might be interesting for investigation and increasing system resilience.
We have Concourse and Vault deployed to EKS cluster with Helm charts. Concourse has workers run in pod, all the Concourse tasks run in docker containers on workers. So it's docker in docker. Once I noticed that one Vault pod placed on same node with Concourse worker.
Worker has overlay FS:
```
overlay 493G 34G 459G 7% /concourse-work-dir/volumes/live/9c8ab166-c87f-44ee-79be-791ba35405f8/volume
overlay 200G 120G 81G 60% /
```
And when Vault start doing raft cluster snaphot:
```
[INFO] storage.raft: starting snapshot up to......
[INFO] storage.raft: compacting logs: from=......
[INFO] storage.raft: snapshot complete up to ....
```
This messages repeated 5-6 times for several minutes and after that one two seconds pod error appeared:
```
[ERROR] core: forward request error: error=\"error during forwarding RPC request\"
```
`show-peers` shows all cluster nodes, leader, but Vault does not respond and all our pipelines failed immediately. cluster CPU utilization dropped to 2-3%. When I run `step-down` command toward cluster CPU Utilization immediately goes up to 50-60%. I tested it several times, result was the same.
To fix it I added `nodeSelector` policy to Vault `StatefullSet` and redeployed vault-0 that was on Concourse worker node. After that system became really stable again.
I have no more logs, only what I provided here. But it really looks like vault cluster were crashed by `overlay` fs.
**To Reproduce**
Put on one kubernetes node Concourse workers with overlay FS and Vault leader node.
Start raft cluster snapshot.
```
kubectl get po,pvc,pv -n vault
kubectl describe pod/concourse-workers-worker-1 -n concourse
```
**Expected behavior**
Vault server to check FS and do not place server on node with overlay FS.
**Environment**
* Kubernetes version:
EKS v1.17.12-eks-7684af
Docker docker://19.3.13
Vault v1.5.4
Concourse v6.7.2
* vault-helm version: 0.8.0
**Additional context**
This is problem and improvement suggestion in one. Feel free to move it to right topic.
Contributor guide
Assessment
This issue has not been assessed yet.