hashicorp / hashicorp/vault-helm

StatefulSet never successfully rolled out when pods restart on new nodes

Open
#795 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Shell
Stars
1.3k
Forks
898
Avg merge
3d 1h
Merged PRs (30d)
1

Description

**Describe the bug**
StatefulSet never successfully rolled out when pods restart on new nodes.

![image](https://user-images.githubusercontent.com/115090256/194092542-c1331bc3-6b3c-4ce5-9bd1-89950dbfc281.png)

- The pods are however well started and the service is operational.
- This is only seen when the pods are moved to new nodes.
- When installing or upgrading this is not seen.

**To Reproduce**
Steps to reproduce the behavior:
1. Install chart
2. Wait for pods to be moved on another node

Other useful info to include: vault pod logs, `kubectl describe statefulset vault` and `kubectl get statefulset vault -o yaml` output:

```
Replicas: 3 desired | 3 total
Update Strategy: OnDelete
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=vault-prod
app.kubernetes.io/name=vault
component=server
helm.sh/chart=vault-0.22.0
```
**This may be related to the recommended OnDelete strategy.**

**Expected behavior**
StatefulSet successfully rolled out when pods restart on new nodes

**Environment**
* Kubernetes version:
* Distribution or cloud vendor: GKE v1.21.14-gke.2700
* vault-helm version: 0.22.0

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the StatefulSet configuration in the vault-helm chart at version 0.22.0 and reproduce the node-move scenario described with Kubernetes. Compare rollout status with the reported pod state and outputs from kubectl describe statefulset vault and kubectl get statefulset vault -o yaml; done means the StatefulSet reports a successful rollout after pods restart on new nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.