Updating StorageClass for PVCs doesn't seem to work as expected

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
• **Where do you run it - cloud or metal? kubernetes (aws/gcp/azure) on various k8s deployment technologies (kops on AWS for this specific instance)
• Are you running Postgres Operator in production? yes
• Type of issue? question, or maybe a bug? Not sure.

Deployment and operation of postgres-operator is good. No major issues noted, except when we attempt to change the storageclass of the PVCs.

It recognizes that the statefulset and pvcs require updates (pvc via the statefulset, not directly?), but doesn't seem to actually apply them (full logs from the update below):

time="2023-09-12T17:51:15Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="volume claims do not require changes" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="syncing statefulsets" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="statefulset qa/mypostgres is not in the desired state and needs to be updated" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-        storageClassName: gp2-encrypted," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+        storageClassName: gp3-encrypted," cluster-name=qa/mypostgres pkg=cluster

Is this expected behavior? It's easy enough to fix (delete replica pvc/pod, wait for sync, failover, delete old leader pvc/pod, wait for sync), but if that's not expected I'd rather ask than assume.

Thanks!

Full update logs:

time="2023-09-12T17:51:15Z" level=info msg="UPDATE event has been queued" cluster-name=qa/mypostgres pkg=controller worker=0
time="2023-09-12T17:51:15Z" level=info msg="update of the cluster started" cluster-name=qa/mypostgres pkg=controller worker=0
time="2023-09-12T17:51:15Z" level=debug msg="-  kind: postgresql," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-  apiVersion: acid.zalan.do/v1," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-    resourceVersion: 4535293469," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-    generation: 9," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+    resourceVersion: 4618120789," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+    generation: 10," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      helm.sh/chart: int-postgresql-1.2.3," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+      helm.sh/chart: int-postgresql-1.2.7," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      storageClass: gp2-encrypted" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+      storageClass: gp3-encrypted" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"10.0.0.0/8\"]" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="final load balancer source ranges as seen in a service spec (not necessarily applied): [\"10.0.0.0/8\"]" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="volume claims do not require changes" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="syncing statefulsets" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="statefulset qa/mypostgres is not in the desired state and needs to be updated" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          resources: {}," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          imagePullPolicy: Always" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+          resources: {}" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      restartPolicy: Always," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      apiVersion: v1," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-        storageClassName: gp2-encrypted," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+        storageClassName: gp3-encrypted," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      status: {" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-        phase: Pending" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-      }" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+      status: {}" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-  }," cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="+  }" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=info msg="reason: new statefulset's volumeClaimTemplates specification for volume \"pgdata\" does not match the current one" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="replacing statefulset" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:15Z" level=debug msg="waiting for the statefulset to be deleted" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="making GET http request: http://10.96.66.38:8008/config" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="making GET http request: http://10.96.56.37:8008/config" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="making GET http request: http://10.96.56.37:8008/patroni" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="making GET http request: http://10.96.66.38:8008/patroni" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="syncing roles" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="closing database connection" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=debug msg="syncing connection pooler (master, replica) from (false, nil) to (false, nil)" cluster-name=qa/mypostgres pkg=cluster
time="2023-09-12T17:51:18Z" level=info msg="cluster has been updated" cluster-name=qa/mypostgres pkg=cluster

After the postgresql.yaml update is deployed:

$ kubectl get pods,pvc,statefulsets -n qa -l application=spilo; echo "STATEFULSET STORAGECLASS"; kubectl describe statefulset mypostgres -n qa | grep gp;

NAME                                    READY   STATUS    RESTARTS   AGE
pod/mypostgres-0   1/1     Running   0          19d
pod/mypostgres-1   1/1     Running   0          19d

NAME                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
persistentvolumeclaim/pgdata-mypostgres-0   Bound    pvc-UUID   500Gi      RWO            gp2-encrypted   2y233d
persistentvolumeclaim/pgdata-mypostgres-1   Bound    pvc-UUID   500Gi      RWO            gp2-encrypted   2y233d

NAME                                               READY   AGE
statefulset.apps/mypostgres   2/2     28m

STATEFULSET STORAGECLASS
  StorageClass:  gp3-encrypted

Start with the volume-syncing and StatefulSet replacement paths described in the logs, focusing on PVCs and StatefulSet volumeClaimTemplates. Reproduce the gp2-encrypted to gp3-encrypted change and compare the desired StatefulSet with the existing PVCs. Done should clarify whether this storage-class change is supported and either document the expected behavior or provide a tested fix.