operator-framework / operator-framework/helm-operator-plugins

Auto-rollback of failed upgrade fails with some resources

Abierto
#421 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Lenguaje dominante
Go
Estrellas
60
Forks
53
Merge medio
6 d 7 h
PR fusionados (30 d)
5

Descripción

It seems that helm's force option for rollbacks (that this library uses here) does not deal well with some resources.

When using the stackrox operator the error looks like this:

rollback failed: failed to replace object: PersistentVolumeClaim "scanner-v4-db" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
      AccessModes:      {"ReadWriteOnce"},
      Selector:         nil,
      Resources:        {Requests: {s"storage": {i: {...}, s: "50Gi", Format: "BinarySI"}}},
-     VolumeName:       "pvc-a78c78d0-a692-47d6-b01a-be7f8d6ebd4a",
+     VolumeName:       "",
-     StorageClassName: &"standard-csi",
+     StorageClassName: nil,
      VolumeMode:       &"Filesystem",
      DataSource:       nil,
      ... // 2 identical fields
  }
: original upgrade error: cannot patch "scanner-v4-matcher-config" with kind ConfigMap:  "" is invalid: patch: Invalid value: "
{
  "apiVersion": "v1",
  "data": {
    "config.yaml": {
      "matcher": {
        "vulnerabilities_url": "https://central.stackrox.svc/api/extensions/scannerdefinitions?version=dev"
      }
    }
  },
  "kind": "ConfigMap",
  "metadata": {
[... snip ...]
  }
}": json: cannot unmarshal object into Go struct field ConfigMap.data of type string 

It may be easier to understand what is going on on the following example, since the issue also happens with helm CLI. It's just that while the CLI warns to exercise caution with this option, the helm-operator-plugins library uses force automatically and unconditionally.

Let's say I have a chart that contains just a PVC such as this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  accessModes:
  - ReadWriteOnce
{{- if .Values.badness }}
  storageClassName:
    nonsense: yes
{{- else }}
  storageClassName: standard
{{- end }}
  resources:
    requests:
      storage: 500Mi

Then I:

helm install --debug repro1 .

This works fine. Now I create any pod that uses the PVC to make it flip to a Bound state.

Then I try to apply a broken upgrade:

$ helm upgrade repro1 . --set badness=true
Error: UPGRADE FAILED: cannot patch "pvc1" with kind PersistentVolumeClaim:  "" is invalid: patch: Invalid value: "{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"repro1\",\"meta.helm.sh/release-namespace\":\"default\",\"pv.kubernetes.io/bind-completed\":\"yes\",\"pv.kubernetes.io/bound-by-controller\":\"yes\",\"volume.beta.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\",\"volume.kubernetes.io/selected-node\":\"kind-control-plane\",\"volume.kubernetes.io/storage-provisioner\":\"rancher.io/local-path\"},\"creationTimestamp\":\"2025-02-13T12:00:25Z\",\"finalizers\":[\"kubernetes.io/pvc-protection\"],\"labels\":{\"app.kubernetes.io/managed-by\":\"Helm\"},\"managedFields\":[{\"manager\":\"helm\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:25Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:meta.helm.sh/release-name\":{},\"f:meta.helm.sh/release-namespace\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/managed-by\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}}},{\"manager\":\"kube-scheduler\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:38Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:volume.kubernetes.io/selected-node\":{}}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:pv.kubernetes.io/bind-completed\":{},\"f:pv.kubernetes.io/bound-by-controller\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{},\"f:volume.kubernetes.io/storage-provisioner\":{}}},\"f:spec\":{\"f:volumeName\":{}}}},{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2025-02-13T12:00:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:accessModes\":{},\"f:capacity\":{\".\":{},\"f:storage\":{}},\"f:phase\":{}}},\"subresource\":\"status\"}],\"name\":\"pvc1\",\"namespace\":\"default\",\"resourceVersion\":\"764382\",\"uid\":\"75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"500Mi\"}},\"storageClassName\":{\"nonsense\":true},\"volumeMode\":\"Filesystem\",\"volumeName\":\"pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62\"},\"status\":{\"phase\":\"Bound\",\"accessModes\":[\"ReadWriteOnce\"],\"capacity\":{\"storage\":\"500Mi\"}}}": json: cannot unmarshal object into Go struct field PersistentVolumeClaimSpec.spec.storageClassName of type string

This is expected. Now in this case the library would try to roll back, but with the force option this fails:

$ helm rollback repro1 1 --force
Error: failed to replace object: PersistentVolumeClaim "pvc1" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "500Mi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-75dd22fc-06ea-4ee0-b9e5-76eb054c3c62",
+ 	VolumeName:       "",
  	StorageClassName: &"standard",
  	VolumeMode:       &"Filesystem",
  	... // 3 identical fields
  }

And then we fall into the loop as described in https://github.com/operator-framework/helm-operator-plugins/issues/224

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Comienza en pkg/client/actionclient.go alrededor de la línea 291 y sigue la ruta de rollback automático, usando el issue 224 enlazado como contexto. Reproduce los fallos de PVC y ConfigMap descritos con los ejemplos de Helm CLI y verifica después que una actualización fallida ya no entre en el bucle de rollback cuando un rollback forzado no pueda restaurar estos recursos.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
go, kubernetes
Área
devops, infrastructure
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.