rustfs / rustfs/helm

Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated

Open Beginner friendly
#16 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
CSS
Stars
24
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Title

Helm chart: Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated

Body

Chart: rustfs (helm/rustfs), version 0.8.0

Problem

templates/deployment.yaml (and templates/statefulset.yaml) load credentials into the
container purely via:

envFrom:
  - secretRef:
      name: {{ include "rustfs.secretName" . }}

Kubernetes only reads envFrom/env values when a Pod is created — it never live-updates a
running container's environment when the referenced Secret's contents change. So if a user
changes secret.rustfs.access_key / secret.rustfs.secret_key (or updates the Secret named by
secret.existingSecret) and runs helm upgrade, Helm patches the Secret object, but the
existing Deployment/StatefulSet pod template hash doesn't change (nothing in spec.template
references the secret content), so no rollout is triggered and the running pod keeps using
the old credentials indefinitely.

values.yaml already declares a podAnnotations: {} value (line ~150), but it's never consumed
by templates/deployment.yaml or templates/statefulset.yaml, so there's no way to wire in the
standard checksum/secret annotation pattern from chart values alone.

Impact

Any credential rotation via helm upgrade silently no-ops on the running pod. In our case this
surfaced as a helm upgrade ... --wait timeout: our own post-upgrade provisioning hook polls the
RustFS S3 API with the new credentials, but the still-running pod only accepts the old ones,
so the hook spins until the wait timeout is hit and the release is reported failed — even though
from RustFS's perspective nothing is actually broken, it's just serving stale credentials.

Suggested fix

Wire .Values.podAnnotations into the pod template in both deployment.yaml and
statefulset.yaml (as most charts do), and/or add a checksum/secret annotation on the pod
template computed from the rendered secret.yaml, e.g.:

template:
  metadata:
    annotations:
      checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
      {{- with .Values.podAnnotations }}
      {{- toYaml . | nindent 8 }}
      {{- end }}

Either would let users trigger a rollout on credential rotation without having to manually
delete the pod.

To reproduce
  1. helm install rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=...
  2. helm upgrade rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=<new value>
  3. kubectl exec into the running pod and check RUSTFS_SECRET_KEY — still the old value, pod age unchanged.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with templates/deployment.yaml, templates/statefulset.yaml, and values.yaml, focusing on the unused podAnnotations value and the secret reference in each pod template. Render or upgrade the chart with changed credentials and verify that the pod template annotations change and Kubernetes triggers a rollout for both credential configuration paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.