Expanding PVC Volume Template Results in Data Loss

When trying to expand the PVC volume template the operator will delete/re-create the PVC volumes instead of just resizing them. We are using Rook-Ceph as the storage provider and have successfully resized PVCs without delete/re-create. We can also manually edit the PVC itself and it will expand. We are using version 0.22.2 of the operator. I've reproduced it in multiple clusters.

We have tried it without the storageManagement options as well and it just results in a loop where the operator will continually try to delete/re-create the PVCs

    storageManagement:
      provisioner: Operator
      reclaimPolicy: Retain

---
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "clickhouse"

spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: default
      podTemplate: clickhouse:23.7.1.2470-alpine
    storageManagement:
      provisioner: Operator
      reclaimPolicy: Retain

  configuration:
    settings:
         # to allow scrape metrics via embedded prometheus protocol
         prometheus/endpoint: /metrics
         prometheus/port: 8888
         prometheus/metrics: true
         prometheus/events: true
         prometheus/asynchronous_metrics: true
    zookeeper:
      nodes:
      - host: clickhouse-keeper.clickhouse.svc.cluster.local
    users:
      default/networks/ip: "::/0"
      default/password: password
      default/profile: default
      # use cluster Pod CIDR for more security
      backup/networks/ip: 0.0.0.0/0
      # PASSWORD=backup_password; echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
      backup/password_sha256_hex: eb94c11d77f46a0290ba8c4fca1a7fd315b72e1e6c83146e42117c568cc3ea4d
    clusters:
      - name: replicated
        layout:
          shardsCount: 1
          replicasCount: 3
    files:
      config.xml: |
          <?xml version="1.0"?>
          <yandex>
            <remote_servers>
                <!-- Test only shard config for testing distributed storage -->
                <ch_cluster>
                    <shard>
                        <internal_replication>True</internal_replication>
                          <replica>
                              <host>chi-clickhouse-replicated-0-0</host>
                              <port>9000</port>
                              <secure>0</secure>
                          </replica>
                          <replica>
                              <host>chi-clickhouse-replicated-0-1</host>
                              <port>9000</port>
                              <secure>0</secure>
                          </replica>
                          <replica>
                              <host>chi-clickhouse-replicated-0-2</host>
                              <port>9000</port>
                              <secure>0</secure>
                          </replica>
                    </shard>
                </ch_cluster>
            </remote_servers>


            <!-- If element has 'incl' attribute, then for it's value will be used corresponding substitution from another file.
                By default, path to file with substitutions is /etc/metrika.xml. It could be changed in config in 'include_from' element.
                Values for substitutions are specified in /clickhouse/name_of_substitution elements in that file.
              -->

            <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
                Optional. If you don't use replicated tables, you could omit that.

                See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
              -->

            <zookeeper>
                <node>
                    <host>clickhouse-keeper.clickhouse.svc.cluster.local</host>
                    <port>2181</port>
                    <secure>0</secure>
                </node>
            </zookeeper>
            <!--
              OpenTelemetry log contains OpenTelemetry trace spans.
            -->
            <opentelemetry_span_log>
              <!--
                  The default table creation code is insufficient, this <engine> spec
                  is a workaround. There is no 'event_time' for this log, but two times,
                  start and finish. It is sorted by finish time, to avoid inserting
                  data too far away in the past (probably we can sometimes insert a span
                  that is seconds earlier than the last span in the table, due to a race
                  between several spans inserted in parallel). This gives the spans a
                  global order that we can use to e.g. retry insertion into some external
                  system.
              -->
              <engine>
                  engine MergeTree
                  partition by toYYYYMM(finish_date)
                  order by (finish_date, finish_time_us, trace_id)
              </engine>
              <database>system</database>
              <table>opentelemetry_span_log</table>
              <flush_interval_milliseconds>7500</flush_interval_milliseconds>
            </opentelemetry_span_log>
          </yandex>
  templates:
    volumeClaimTemplates:
      - name: default
        spec:
          accessModes:
            - ReadWriteOnce
          reclaimPolicy: Retain
          resources:
            requests:
              storage: 55Gi
    podTemplates:
      - name: clickhouse:23.7.1.2470-alpine
        metadata:
          annotations:
              prometheus.io/scrape: 'true'
              prometheus.io/port: '8888'
              prometheus.io/path: '/metrics'
              # need separate prometheus scrape config, look to https://github.com/prometheus/prometheus/issues/3756
              clickhouse.backup/scrape: 'true'
              clickhouse.backup/port: '7171'
              clickhouse.backup/path: '/metrics'
        spec:
          containers:
            - name: clickhouse-pod
              image: clickhouse-server:23.7.1.2470-alpine
            - name: clickhouse-backup
              image: clickhouse-backup:latest
              imagePullPolicy: Always
              command:
                - bash
                - -xc
                - "/bin/clickhouse-backup server"
              env:
                - name: CLICKHOUSE_PASSWORD
                  value: password
                - name: LOG_LEVEL
                  value: "debug"
                - name: ALLOW_EMPTY_BACKUPS
                  value: "true"
                - name: API_LISTEN
                  value: "0.0.0.0:7171"
                # INSERT INTO system.backup_actions to execute backup
                - name: API_CREATE_INTEGRATION_TABLES
                  value: "true"
                - name: BACKUPS_TO_KEEP_REMOTE
                  value: "3"
                # change it for production S3
                - name: REMOTE_STORAGE
                  value: "s3"
                - name: S3_ACL
                  value: "private"
                - name: S3_ENDPOINT
                  value: https://minio
                - name: S3_BUCKET
                  value: clickhouse-backups
                # {shard} macro defined by clickhouse-operator
                - name: S3_PATH
                  value: backup/shard-{shard}
                - name: S3_ACCESS_KEY
                  value: clickhouse_backups_rw
                - name: S3_DISABLE_CERT_VERIFICATION
                  value: "true"
                - name: S3_SECRET_KEY
                  value: password
                - name: S3_FORCE_PATH_STYLE
                  value: "true"
              ports:
                - name: backup-rest
                  containerPort: 7171

No source files or tests are named. Start by reproducing PVC volume-template expansion with the provided ClickHouseInstallation on operator version 0.22.2, using Rook-Ceph and the shown storageManagement settings. Done means existing PVCs are resized in place rather than deleted and recreated, and the operator no longer loops when those settings are omitted.