Cluster service deleted on upgrade due to reconcile failure

While performing an upgrade via Helm from 0.23.2 to 0.23.6, I ran across a problem where the cluster service disappeared. I also included a minor upgrade of the altinitystable image, but I don't think that is related.

The important bits in my CHI resource:

spec:
  defaults:
    templates:
      podTemplate: default-clickhouse-pod
      dataVolumeClaimTemplate: default-data-volume
      logVolumeClaimTemplate: default-log-volume
      clusterServiceTemplate: default-service-template
  configuration:
    settings:
      logger/level: information
    clusters:
      - name: events
        layout:
          shardsCount: 1
          replicasCount: 3
        secret:
          auto: "true"
  templates:
    serviceTemplates:
      - name: default-service-template
        generateName: clickhouse-{chi}
        metadata:
          annotations:
            cloud.google.com/load-balancer-type: "Internal"
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
            service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
        spec:
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
          type: LoadBalancer

When the operator upgraded, it appeared to get stuck attempting to convert clickhouse-events from a LoadBalancer to a ClusterIP. I believe this is somehow related to this commit that changes the default from LoadBalancer to ClusterIP. However, this CHI has always explicitly set the template to use LoadBalancer.

On startup, I saw this in the logs:

I0710 05:05:48.675757       1 service.go:86] CreateServiceCluster():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:foo/clickhouse-events
I0710 05:05:48.676889       1 worker-chi-reconciler.go:907] reconcileService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:Service: foo/clickhouse-events not found. err: service "clickhouse-events" not found
I0710 05:05:48.840035       1 deleter.go:322] deleteServiceIfExists():foo/clickhouse-events:Not Found Service: foo/clickhouse-events err: services "clickhouse-events" not found
I0710 05:05:49.062109       1 worker.go:1480] createService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:OK Create Service: foo/clickhouse-events
I0710 05:05:49.883043       1 worker-chi-reconciler.go:922] reconcileService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:Service reconcile successful: foo/clickhouse-events

...

I0710 05:06:25.213119       1 worker-chi-reconciler.go:900] reconcileService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:Service found: foo/clickhouse-events. Will try to update
E0710 05:06:25.213168       1 worker-chi-reconciler.go:914] reconcileService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:Update Service: foo/clickhouse-events failed with error: just recreate the service in case of service type change 'LoadBalancer'=>'ClusterIP'
I0710 05:06:26.384478       1 deleter.go:329] deleteServiceIfExists():foo/clickhouse-events:OK delete Service: foo/clickhouse-events
E0710 05:06:26.584816       1 worker.go:1486] createService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:FAILED Create Service: foo/clickhouse-events err: object is being deleted: services "clickhouse-events" already exists
E0710 05:06:27.422151       1 worker-chi-reconciler.go:928] reconcileService():foo/events/c857c1dd-66bf-4182-96eb-3e45f61664ee:FAILED to reconcile Service: foo/clickhouse-events CHI: events

It now appears to be recreated on a forced restart of the operator, and then a minute or so later, is deleted again. It won't be recreated until the operator restarts again.

I0710 05:16:25.276854       1 service.go:86] CreateServiceCluster():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:foo/clickhouse-events
I0710 05:16:25.278246       1 worker-chi-reconciler.go:907] reconcileService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:Service: foo/clickhouse-events not found. err: service "clickhouse-events" not found
I0710 05:16:25.435511       1 deleter.go:322] deleteServiceIfExists():foo/clickhouse-events:Not Found Service: foo/clickhouse-events err: services "clickhouse-events" not found
I0710 05:16:25.805221       1 worker.go:1480] createService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:OK Create Service: foo/clickhouse-events
I0710 05:16:26.468825       1 worker-chi-reconciler.go:922] reconcileService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:Service reconcile successful: foo/clickhouse-events

...

I0710 05:17:26.904518       1 worker-chi-reconciler.go:900] reconcileService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:Service found: foo/clickhouse-events. Will try to update
E0710 05:17:26.904648       1 worker-chi-reconciler.go:914] reconcileService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:Update Service: foo/clickhouse-events failed with error: just recreate the service in case of service type change 'LoadBalancer'=>'ClusterIP'
I0710 05:17:28.073703       1 deleter.go:329] deleteServiceIfExists():foo/clickhouse-events:OK delete Service: foo/clickhouse-events
E0710 05:17:28.274057       1 worker.go:1486] createService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:FAILED Create Service: foo/clickhouse-events err: object is being deleted: services "clickhouse-events" already exists
E0710 05:17:29.119358       1 worker-chi-reconciler.go:928] reconcileService():foo/events/a84245ce-80bc-4fb9-ad58-c69a77019a70:FAILED to reconcile Service: foo/clickhouse-events CHI: events 

Note: When it is creates, it is created correctly as a LoadBalancer, but then the second resource reconciliation attempts to make it a ClusterIP again.

Start with the service reconciliation flow in worker-chi-reconciler.go, then inspect service.go, worker.go, and deleter.go around the logged create, update, delete, and recreate operations. Reproduce the Helm upgrade from 0.23.2 to 0.23.6 with the LoadBalancer service template and trace why the second reconciliation resolves it as ClusterIP. Done means the explicitly configured LoadBalancer service survives reconciliation without deletion or an object-being-deleted failure.