operator-framework / operator-framework/helm-operator-plugins
Auto-rollback of failed upgrade fails with some resources
Nessuno ha ancora preso questa issue.
- Lingua principale
- Go
- Stelle
- 60
- Fork
- 53
- Merge medio
- 6g 7h
- PR unite (30g)
- 5
Descrizione
It seems that helm's force option for rollbacks (that this library uses here) does not deal well with some resources.
When using the stackrox operator the error looks like this:
rollback failed: failed to replace object: PersistentVolumeClaim "scanner-v4-db" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
core.PersistentVolumeClaimSpec{
AccessModes: {"ReadWriteOnce"},
Selector: nil,
Resources: {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}},
- VolumeName: "pvc-a78c78d0-a692-47d6-b01a-be7f8d6ebd4a",
+ VolumeName: "",
- StorageClassName: &"standard-csi",
+ StorageClassName: nil,
VolumeMode: &"Filesystem",
DataSource: nil,
... // 2 identical fields
}
: original upgrade error: cannot patch "scanner-v4-matcher-config" with kind ConfigMap: "" is invalid: patch: Invalid value: "
{
"apiVersion": "v1",
"data": {
"config.yaml": {
"matcher": {
"vulnerabilities_url": "https://central.stackrox.svc/api/extensions/scannerdefinitions?version=dev"
}
}
},
"kind": "ConfigMap",
"metadata": {
[... snip ...]
}
}": json: cannot unmarshal object into Go struct field ConfigMap.data of type string
It may be easier to understand what is going on on the following example, since the issue also happens with helm CLI. It's just that while the CLI warns to exercise caution with this option, the helm-operator-plugins library uses force automatically and unconditionally.
Let's say I have a chart that contains just a PVC such as this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc1
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.badness }}
storageClassName:
nonsense: yes
{{- else }}
storageClassName: standard
{{- end }}
resources:
requests:
storage: 500Mi
Then I:
helm install --debug repro1 .
This works fine. Now I create any pod that uses the PVC to make it flip to a Bound state.
Then I try to apply a broken upgrade:
$ helm upgrade repro1 . --set badness=true
Error: UPGRADE FAILED: cannot patch "pvc1" with kind PersistentVolumeClaim: "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"repro1\",\"meta.helm.sh/release-namespace\":\"default\",\"pv.kubernetes.io/bind-completed\":\"yes\",\"pv.kubernetes.io/bound-by-controller\":\"yes\",\"volume.beta.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\",\"volume.kubernetes.io/selected-node\":\"kind-control-plane\",\"volume.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\"},\"creationTimestamp\":\"2025-02-13T12:00:25Z\",\"finalizers\":[\"kubernetes.io/pvc-protection\"],\"labels\":{\"app.kubernetes.io/managed-by\":\"Helm\"},\"managedFields\":[{\"manager\":\"helm\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:25Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}}},{\"manager\":\"kube-scheduler\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:38Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:volume.kubernetes.io/selected-node\":{}}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:pv.kubernetes.io/bind-completed\":{},\"f:pv.kubernetes.io/bound-by-controller\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{},\"f:volume.kubernetes.io/storage-provisioner\":{}}},\"f:spec\":{\"f:volumeName\":{}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:accessModes\":{},\"f:capacity\":{\".\":{},\"f:storage\":{}},\"f:phase\":{}}},\"subresource\":\"status\"}],\"name\":\"pvc1\",\"namespace\":\"default\",\"resourceVersion\":\"764382\",\"uid\":\"75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"500Mi\"}},\"storageClassName\":{\"nonsense\":true},\"volumeMode\":\"Filesystem\",\"volumeName\":\"pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"status\":{\"phase\":\"Bound\",\"accessModes\":[\"ReadWriteOnce\"],\"capacity\":{\"storage\":\"500Mi\"}}}": json: cannot unmarshal object into Go struct field PersistentVolumeClaimSpec.spec.storageClassName of type string
This is expected. Now in this case the library would try to roll back, but with the force option this fails:
$ helm rollback repro1 1 --force
Error: failed to replace object: PersistentVolumeClaim "pvc1" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
core.PersistentVolumeClaimSpec{
AccessModes: {"ReadWriteOnce"},
Selector: nil,
Resources: {Requests: {s"storage": {i: {...}, s: "500Mi", Format: "BinarySI"}}},
- VolumeName: "pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62",
+ VolumeName: "",
StorageClassName: &"standard",
VolumeMode: &"Filesystem",
... // 3 identical fields
}
And then we fall into the loop as described in https://github.com/operator-framework/helm-operator-plugins/issues/224
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia in pkg/client/actionclient.go intorno alla riga 291 e segui il percorso di rollback automatico, usando come contesto l’issue 224 collegata. Riproduci gli errori di PVC e ConfigMap descritti con gli esempi di Helm CLI, quindi verifica che un upgrade non riuscito non entri più nel ciclo di rollback quando un rollback forzato non riesce a ripristinare queste risorse.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- go, kubernetes
- Ambito
- devops, infrastructure
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100