Can't revert sidecar containers and extra volumes on vitess operator

We have an issue in our vitess operator deployment. We are running vitess/lite:v12.0.0-mysql80. I added in the operator two sidecar containers and two extra volume to the gateway section. It looks like that -

  cells:
  - name: {{ app_cell }}
    gateway:
      sidecarContainers:
      - image: docker.elastic.co/beats/filebeat:8.3.1
        name: filebeat
        args: [
            "-c", "/etc/filebeat.yml",
            "-e",
        ]
        env:
        - name: LOGSTASH_URL
          value: "XXX"
        volumeMounts:
        - name: filebeat-config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: logs-dir
          mountPath: /var/logs/vitess
          readOnly: true
      - image: blacklabelops/logrotate:1.3
        name: logrotate
        volumeMounts:
        - name: logs-dir
          mountPath: /var/logs/vitess
        lifecycle:
          postStart:
            exec:
              command:
              - sh
              - -c
              - echo "XXX"
        env:
        - name: LOGS_DIRECTORIES
          value: "/var/logs/vitess"
        - name: LOG_FILE_ENDINGS
          value: "json"
        - name: LOGROTATE_CRONSCHEDULE
          value: "*/1 * * * *"
        - name: LOGROTATE_COPIES
          value: "2"
        - name: LOGROTATE_SIZE
          value: "50M"
      extraVolumes:
      - name: logs-dir
        emptyDir:
          medium: "Memory"
          sizeLimit: "0.5Gi"
      - name: filebeat-config
        configMap:
          name: vitess-filebeat-config
      extraVolumeMounts:
      - name: logs-dir
        mountPath: /var/logs/vitess

All of the changes are applied nicely, but once I applied them, I can’t revert to the previous configuration. Applying the yaml again without the sidecar containers and the extra volumes doesn’t do anything in itself, and I’m getting this error message in the operator pod:

{
  "level": "error",
  "ts": XXX,
  "logger": "controller.vitesscell-controller",
  "msg": "Reconciler error",
  "name": "vitess-app-XXX",
  "namespace": "vitess",
  "error": "Deployment.apps \"vitess-app-vtgate-XXX\" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"filebeat-config\", spec.template.spec.containers[1].volumeMounts[1].name: Not found: \"logs-dir\", spec.template.spec.containers[2].volumeMounts[0].name: Not found: \"logs-dir\"]",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:214"
}

The only way I found to apply the changes was to delete the vitess-app-vtgate deployment - and obviously I don’t wanna do that in my production environment.

Can you help?

Start with the controller.vitesscell-controller reconciliation that updates the vitess-app-vtgate Deployment, using the supplied manifest and validation error as the reproduction. Trace what happens when sidecarContainers, extraVolumes, and extraVolumeMounts are removed, then verify that reconciliation produces a valid Deployment without requiring deletion.