PV build up with Reclaim policy set Delete

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.0

Helm Chart Version

0.22.0

CertManager Version

1.10.1

Deployment Method

Helm

cert-manager installation

Yes I have installed cert manager following the steps mentioned in documentation.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example-1
spec:
  replicas: 1
  organization: xyz
  labels: 
    - arc-1
    - linux
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: docker
        volumeMounts:
        - name: var-lib-docker
          mountPath: /var/lib/docker
  volumeClaimTemplates:
  - metadata:
      name: var-lib-docker
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 7Gi
      storageClassName: gh-ebs
      dataSource:
        name: ebs-volume-snapshot
        kind: VolumeSnapshot
        apiGroup: snapshot.storage.k8s.io
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gh-ebs
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

To Reproduce

1. Install ARC.
2. Start a runnerSet with volumeClaimTemplate.
3. Run a couple of workflows.
4. Observe that PVs with ReclaimPolicy delete are building up despite PVCs being deleted.

Describe the bug

Dynamically provisioned Persistent Volumes that are in an available state are unable to cleaned up by EBS CSI with the error that the volume is still attached to the node.
Example log : delete "pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a": volume deletion failed: persistentvolume pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a is still attached to node ip-10-10-2-152.eu-central-1.compute.internal.

Describe the expected behavior

Dynamically provisioned persistent volumes with ReclaimPolicy set to Delete should be deleted when PVC is deleted.

Whole Controller Logs

2023-02-06T12:33:19Z	DEBUG	runnerpersistentvolume	Retrying sync until pvc gets released	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a", "requeueAfter": "10s"}
2023-02-06T12:33:19Z	ERROR	Reconciler error	{"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"var-lib-docker-nitro-1-5d5sx-0","namespace":"actions-runner-system"}, "namespace": "actions-runner-system", "name": "var-lib-docker-example-1-5d5sx-0", "reconcileID": "7aeac10f-6998-430e-8c2a-adc94b385299", "error": "Operation cannot be fulfilled on persistentvolumes \"pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a\": the object has been modified; please apply your changes to the latest version and try again"}
2023-02-06T12:33:19Z	INFO	runnerpersistentvolume	PV should be Available now	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a"}
2023-02-06T14:29:08Z	DEBUG	runnerpersistentvolume	Retrying sync until pvc gets released	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a", "requeueAfter": "10s"}
2023-02-06T14:29:08Z	INFO	runnerpersistentvolume	PV should be Available now	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a"}
2023-02-06T14:32:22Z	DEBUG	runnerpersistentvolume	Retrying sync until pvc gets released	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a", "requeueAfter": "10s"}
2023-02-06T14:32:22Z	INFO	runnerpersistentvolume	PV should be Available now	{"pv": "/pvc-df682ae3-3b7b-4599-bdce-e9b17dda2a7a"}

Whole Runner Pod Logs

Not really related to runner logs

Additional Context

I suspect the issue is because of pending finalizer [kubernetes.io/pv-protection] on the PV. Deleting the Persistent volumes in Kubernetes does not delete the AWS EBS volumes.

Start with the runnerpersistentvolume reconciliation reported in the controller logs, and inspect how the runnerpersistentvolumeclaim-controller handles PV finalizers and release state. Reproduce the RunnerSet with volumeClaimTemplates and reclaimPolicy Delete, then verify that deleting the PVC also removes the PV and its dynamically provisioned EBS volume without repeated retry logs.