Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated
Nobody has claimed this yet.
- Dominant language
- CSS
- Stars
- 24
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Title
Helm chart: Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated
Body
Chart: rustfs (helm/rustfs), version 0.8.0
Problem
templates/deployment.yaml (and templates/statefulset.yaml) load credentials into the
container purely via:
envFrom:
- secretRef:
name: {{ include "rustfs.secretName" . }}
Kubernetes only reads envFrom/env values when a Pod is created — it never live-updates a
running container's environment when the referenced Secret's contents change. So if a user
changes secret.rustfs.access_key / secret.rustfs.secret_key (or updates the Secret named by
secret.existingSecret) and runs helm upgrade, Helm patches the Secret object, but the
existing Deployment/StatefulSet pod template hash doesn't change (nothing in spec.template
references the secret content), so no rollout is triggered and the running pod keeps using
the old credentials indefinitely.
values.yaml already declares a podAnnotations: {} value (line ~150), but it's never consumed
by templates/deployment.yaml or templates/statefulset.yaml, so there's no way to wire in the
standard checksum/secret annotation pattern from chart values alone.
Impact
Any credential rotation via helm upgrade silently no-ops on the running pod. In our case this
surfaced as a helm upgrade ... --wait timeout: our own post-upgrade provisioning hook polls the
RustFS S3 API with the new credentials, but the still-running pod only accepts the old ones,
so the hook spins until the wait timeout is hit and the release is reported failed — even though
from RustFS's perspective nothing is actually broken, it's just serving stale credentials.
Suggested fix
Wire .Values.podAnnotations into the pod template in both deployment.yaml and
statefulset.yaml (as most charts do), and/or add a checksum/secret annotation on the pod
template computed from the rendered secret.yaml, e.g.:
template:
metadata:
annotations:
checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
Either would let users trigger a rollout on credential rotation without having to manually
delete the pod.
To reproduce
helm install rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=...helm upgrade rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=<new value>kubectl execinto the running pod and checkRUSTFS_SECRET_KEY— still the old value, pod age unchanged.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with templates/deployment.yaml, templates/statefulset.yaml, and values.yaml, focusing on the unused podAnnotations value and the secret reference in each pod template. Render or upgrade the chart with changed credentials and verify that the pod template annotations change and Kubernetes triggers a rollout for both credential configuration paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100