Vttablets are in a restart loop when default ServiceAccount uses imagePullSecrets

Hi all,

we happen to use imagePullSecrets in our namespaces.
If we deploy the operator and any VitessCluster, e.g. the 101_initial_cluster.yaml, the vttablet pods get restarted over and over again.

Currently, we are on vitess-14 - I tested various other versions as well with the same problem.

Setup:

vitess operator: 2.7.1
vitess images: vitess/lite:v14.0.0-mysql80
kubernetes: v1.21.4

How to reproduce:

# create namespace
kubectl create ns vitesstest

# deploy operator, e.g. https://github.com/vitessio/vitess/blob/main/examples/operator/operator.yaml
kubectl -n vitesstest apply -f operator.yaml

# create pullSecret
{ cat | kubectl -n vitesstest apply -f -; } << EOF
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXIuZXhhbXBsZS5jb20iOnsiYXV0aCI6ImVXOTFZWEpsT21OMWNtbHZkWE1LIn19fQo=
kind: Secret
metadata:
  name: pullsecret
type: kubernetes.io/dockerconfigjson
EOF

# patch ServiceAccount to use pullSecret
kubectl -n vitesstest patch serviceaccount default -p '{"imagePullSecrets": [{"name": "pullsecret"}]}'

# deploy testcluster
kubectl -n vitesstest apply -f 101_initial_cluster.yaml

Now you see, that the vttablets get restarted over and over again.
If you look at the vitessshards: kubectl -n vitesstest get vitessshards.planetscale.com -o yaml
you see:

[...]
    tablets:
      zone1-2469782763:
        [...]
        pendingChanges: |
          spec:
            imagePullSecrets: null
        [...]
[...]

And the operator tells you:

{"level":"error","ts":1657017915.226152,"logger":"controller.vitessshard-controller","msg":"Reconciler error","name":"example-commerce-x-x-0f5afee6","namespace":"vitesstest","error":"Operation cannot be fulfilled on pods \"example-vttablet-zone1-2469782763-bfadd780\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.7/pkg/internal/controller/controller.go:214"}

For now, I worked around this issue and don't use imagePullSecrets in my vitess namespace but it would be nice if this could be fixed.

Thanks!
chris

Start by reproducing the issue with operator.yaml, 101_initial_cluster.yaml, and the ServiceAccount imagePullSecrets patch. Inspect the VitessShard pendingChanges and controller logs while the vttablet restarts; done means the cluster reconciles without repeated restarts or a spec.imagePullSecrets: null pending change.