PVC migration fails when migrating to csi

pv-migrator error out when copying to csi storage class setup on same blockpool

Here is the ceph version and rook version:

$ ceph version
```
ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
```
rook version: 1.7.11

Here is the non-csi Storageclass definition:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"rook-ceph-block"},"parameters":{"blockPool":"blockpool001","clusterNamespace":"rook-ceph","fstype":"xfs"},"provisioner":"ceph.rook.io/block"}
creationTimestamp: "2022-05-06T19:51:18Z"
name: rook-ceph-block
resourceVersion: "218730968"
uid: bbc5ad00-3e59-4e3f-9363-3708e5024d08
parameters:
blockPool: blockpool001
clusterNamespace: rook-ceph
fstype: xfs
provisioner: ceph.rook.io/block
reclaimPolicy: Delete
volumeBindingMode: Immediate
```

Here is CSI StorageClass:

```
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"rook-ceph-block-csi"},"parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/controller-expand-secret-name":"rook-csi-rbd-provisioner","csi.storage.k8s.io/controller-expand-secret-namespace":"rook-ceph","csi.storage.k8s.io/fstype":"xfs","csi.storage.k8s.io/node-stage-secret-name":"rook-csi-rbd-node","csi.storage.k8s.io/node-stage-secret-namespace":"rook-ceph","csi.storage.k8s.io/provisioner-secret-name":"rook-csi-rbd-provisioner","csi.storage.k8s.io/provisioner-secret-namespace":"rook-ceph","imageFeatures":"layering","imageFormat":"2","pool":"blockpool001"},"provisioner":"rook-ceph-system.rbd.csi.ceph.com","reclaimPolicy":"Delete"}
creationTimestamp: "2023-08-02T21:40:19Z"
name: rook-ceph-block-csi
resourceVersion: "473173630"
uid: a724514f-951c-43fd-a123-a049538f9ab5
parameters:
clusterID: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: xfs
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
imageFeatures: layering
imageFormat: "2"
pool: blockpool001
provisioner: rook-ceph-system.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
```

Error:
```
$ pv-migrator --pvc=prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 --pvc-ns=fabric --destination-sc=rook-ceph-block-csi --rook-ns=rook-ceph-system
I0803 19:22:04.686852 25077 log.go:34] Create Kubernetes Client
I0803 19:22:04.713324 25077 log.go:34] List all the PVC from the source storageclass
I0803 19:22:04.716064 25077 log.go:34] 1 PVCs found with source StorageClass
I0803 19:22:04.716097 25077 log.go:34] Start Migration of PVCs to CSI
I0803 19:22:04.716111 25077 log.go:34] migrating PVC "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2" from namespace "sf-fabric"
I0803 19:22:04.716122 25077 log.go:34] Fetch PV information from PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2
I0803 19:22:04.720047 25077 log.go:34] PV found "pvc-41e06349-b07c-4662-941c-f38d914eabc4"
I0803 19:22:04.720068 25077 log.go:34] Update Reclaim policy from Delete to Reclaim for PV: pvc-41e06349-b07c-4662-941c-f38d914eabc4
I0803 19:22:04.725395 25077 log.go:34] Retrieving old ceph volume name from PV object: pvc-41e06349-b07c-4662-941c-f38d914eabc4
I0803 19:22:04.725416 25077 log.go:34] rbd image name is "pvc-41e06349-b07c-4662-941c-f38d914eabc4"
I0803 19:22:04.725420 25077 log.go:34] Deleting pvc object: prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2
I0803 19:22:04.729446 25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{161061273600 0} {} 150Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (0 seconds elapsed)
I0803 19:22:06.732848 25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 in state &PersistentVolumeClaimStatus{Phase:Bound,AccessModes:[ReadWriteOnce],Capacity:ResourceList{storage: {{161061273600 0} {} 150Gi BinarySI},},Conditions:[]PersistentVolumeClaimCondition{},} to be deleted (2 seconds elapsed)
I0803 19:22:06.736645 25077 log.go:34] Generate new PVC with same name in destination storageclass
I0803 19:22:06.736736 25077 log.go:34] Create new csi pvc
I0803 19:22:06.744044 25077 log.go:34] Waiting up to &PersistentVolumeClaim{ObjectMeta:{prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 sf-fabric 41e06349-b07c-4662-941c-f38d914eabc4 0 2023-08-03 19:21:23 +0000 UTC map[app:prometheus app.kubernetes.io/instance:fabric-layer-1 app.kubernetes.io/managed-by:prometheus-operator app.kubernetes.io/name:prometheus app.kubernetes.io/version:2.27.1 operator.prometheus.io/name:fabric-layer-1 operator.prometheus.io/shard:0 prometheus:fabric-layer-1] map[] [] [kubernetes.io/pvc-protection] [{kube-controller-manager Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{},"f:volume.kubernetes.io/storage-provisioner":{}}},"f:spec":{"f:volumeName":{}}}} {kube-controller-manager Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}} {pv-migrator Update v1 2023-08-03 19:21:23 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"kubernetes.io/pvc-protection\"":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/version":{},"f:operator.prometheus.io/name":{},"f:operator.prometheus.io/shard":{},"f:prometheus":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}}}]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{161061273600 0} {} 150Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*rook-ceph-block-csi,VolumeMode:*Filesystem,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state
I0803 19:22:06.744078 25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 (0 seconds elapsed)
I0803 19:22:08.748111 25077 log.go:34] waiting for PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 (2 seconds elapsed)
I0803 19:22:08.753034 25077 log.go:34] Waiting for PV "pvc-4566e992-8290-4787-b4ea-5c7b61da6634" to bind to PVC "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2"
I0803 19:22:08.753070 25077 log.go:34] Waiting up to 30ms for PersistentVolumeClaims [prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2] to have phase Bound
I0803 19:22:08.755251 25077 log.go:34] PersistentVolumeClaim prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 found and phase=Bound (2.162732ms)
I0803 19:22:08.755269 25077 log.go:34] Waiting up to 3s for PersistentVolume pvc-4566e992-8290-4787-b4ea-5c7b61da6634 to have phase Bound
I0803 19:22:08.757184 25077 log.go:34] PersistentVolume pvc-4566e992-8290-4787-b4ea-5c7b61da6634 found and phase=Bound (1.905595ms)
I0803 19:22:08.764451 25077 log.go:34] New PVC with same name "prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2" created via CSI
I0803 19:22:08.764467 25077 log.go:34] Extracting new volume name from CSI PV
I0803 19:22:08.764486 25077 log.go:34] CSI new volume name: csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa
I0803 19:22:08.764500 25077 log.go:34] Fetching csi pool name
I0803 19:22:08.764517 25077 log.go:34] csi poolname: blockpool001
I0803 19:22:08.764529 25077 log.go:34] Create new Ceph connection
I0803 19:22:08.764541 25077 log.go:34] csi poolname: blockpool001
I0803 19:22:08.766817 25077 log.go:34] clusterID: rook-ceph, monitors: 10.3.238.27:6789,10.3.159.40:6789,10.3.237.157:6789,10.3.213.17:6789,10.3.215.174:6789, poolname: blockpool001
I0803 19:22:08.769419 25077 log.go:34] New connection arg monitors: 10.3.238.27:6789,10.3.159.40:6789,10.3.237.157:6789,10.3.213.17:6789,10.3.215.174:6789, id: csi-rbd-provisioner, keyfile: /tmp/csi/keys/keyfile-779656205, pool: blockpool001, datapool:
I0803 19:22:08.769554 25077 log.go:34] Cluster connection created
I0803 19:22:08.769570 25077 log.go:34] Delete the placeholder CSI volume in ceph cluster
I0803 19:22:15.784896 25077 log.go:34] Successfully removed volume csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa
I0803 19:22:15.784920 25077 log.go:34] Rename old ceph volume to new CSI volume
Error: failed to migrate PVC prometheus-fabric-layer-1-db-prometheus-fabric-layer-1-2 : failed to rename old ceph volume pvc-41e06349-b07c-4662-941c-f38d914eabc4 to new CSI volume csi-vol-08d0af44-3233-11ee-94b3-2e7d69f465aa: exit status 2. failed to rename rbd image, command output: rbd: rename error: (2) No such file or directory
2023-08-03T19:22:15.828+0000 7ff562ead2c0 -1 librbd: error opening source image: (2) No such file or directory
```

What could be the issue? How do I troubleshoot?