operator-framework / operator-framework/helm-operator-plugins
Auto-rollback of failed upgrade fails with some resources
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Go
- Sterne
- 60
- Forks
- 53
- Ø Merge
- 6 T. 7 Std.
- Gemergte PRs (30 T.)
- 5
Beschreibung
It seems that helm's force option for rollbacks (that this library uses here) does not deal well with some resources.
When using the stackrox operator the error looks like this:
rollback failed: failed to replace object: PersistentVolumeClaim "scanner-v4-db" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
core.PersistentVolumeClaimSpec{
AccessModes: {"ReadWriteOnce"},
Selector: nil,
Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}},
- VolumeName: "pvc-a78c78d0-a692-47d6-b01a-be7f8d6ebd4a",
+ VolumeName: "",
- StorageClassName: &"standard-csi",
+ StorageClassName: nil,
VolumeMode: &"Filesystem",
DataSource: nil,
... // 2 identical fields
}
: original upgrade error: cannot patch "scanner-v4-matcher-config" with kind ConfigMap: "" is invalid: patch: Invalid value: "
{
"apiVersion": "v1",
"data": {
"config.yaml": {
"matcher": {
"vulnerabilities_url": "https://central.stackrox.svc/api/extensions/scannerdefinitions?version=dev"
}
}
},
"kind": "ConfigMap",
"metadata": {
[... snip ...]
}
}": json: cannot unmarshal object into Go struct field ConfigMap.data of type string
It may be easier to understand what is going on on the following example, since the issue also happens with helm CLI. It's just that while the CLI warns to exercise caution with this option, the helm-operator-plugins library uses force automatically and unconditionally.
Let's say I have a chart that contains just a PVC such as this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc1
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.badness }}
storageClassName:
nonsense: yes
{{- else }}
storageClassName: standard
{{- end }}
resources:
requests:
storage: 500Mi
Then I:
helm install --debug repro1 .
This works fine. Now I create any pod that uses the PVC to make it flip to a Bound state.
Then I try to apply a broken upgrade:
$ helm upgrade repro1 . --set badness=true
Error: UPGRADE FAILED: cannot patch "pvc1" with kind PersistentVolumeClaim: "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"repro1\",\"meta.helm.sh/release-namespace\":\"default\",\"pv.kubernetes.io/bind-completed\":\"yes\",\"pv.kubernetes.io/bound-by-controller\":\"yes\",\"volume.beta.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\",\"volume.kubernetes.io/selected-node\":\"kind-control-plane\",\"volume.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\"},\"creationTimestamp\":\"2025-02-13T12:00:25Z\",\"finalizers\":[\"kubernetes.io/pvc-protection\"],\"labels\":{\"app.kubernetes.io/managed-by\":\"Helm\"},\"managedFields\":[{\"manager\":\"helm\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:25Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}}},{\"manager\":\"kube-scheduler\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:38Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:volume.kubernetes.io/selected-node\":{}}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:pv.kubernetes.io/bind-completed\":{},\"f:pv.kubernetes.io/bound-by-controller\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{},\"f:volume.kubernetes.io/storage-provisioner\":{}}},\"f:spec\":{\"f:volumeName\":{}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:accessModes\":{},\"f:capacity\":{\".\":{},\"f:storage\":{}},\"f:phase\":{}}},\"subresource\":\"status\"}],\"name\":\"pvc1\",\"namespace\":\"default\",\"resourceVersion\":\"764382\",\"uid\":\"75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"500Mi\"}},\"storageClassName\":{\"nonsense\":true},\"volumeMode\":\"Filesystem\",\"volumeName\":\"pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"status\":{\"phase\":\"Bound\",\"accessModes\":[\"ReadWriteOnce\"],\"capacity\":{\"storage\":\"500Mi\"}}}": json: cannot unmarshal object into Go struct field PersistentVolumeClaimSpec.spec.storageClassName of type string
This is expected. Now in this case the library would try to roll back, but with the force option this fails:
$ helm rollback repro1 1 --force
Error: failed to replace object: PersistentVolumeClaim "pvc1" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
core.PersistentVolumeClaimSpec{
AccessModes: {"ReadWriteOnce"},
Selector: nil,
Resources: {Requests: {s"storage": {i: {...}, s: "500Mi", Format: "BinarySI"}}},
- VolumeName: "pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62",
+ VolumeName: "",
StorageClassName: &"standard",
VolumeMode: &"Filesystem",
... // 3 identical fields
}
And then we fall into the loop as described in https://github.com/operator-framework/helm-operator-plugins/issues/224
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginne in pkg/client/actionclient.go ungefähr bei Zeile 291 und verfolge den Pfad für das automatische Rollback unter Verwendung des verknüpften Issue 224 als Kontext. Reproduziere die beschriebenen PVC- und ConfigMap-Fehler mit den Helm-CLI-Beispielen und überprüfe anschließend, dass ein fehlgeschlagenes Upgrade nicht mehr in die Rollback-Schleife eintritt, wenn ein erzwungenes Rollback diese Ressourcen nicht wiederherstellen kann.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- go, kubernetes
- Bereich
- devops, infrastructure
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100