PVs created from RunnerSet are not reused - results in PVs buildup

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.22.0

Helm Chart Version

v0.7.2

CertManager Version

v1.6.1

Deployment Method

Helm

cert-manager installation

resource "helm_release" "cert_manager" {
  name             = "cert-manager"
  repository       = "https://charts.jetstack.io"
  chart            = "cert-manager"
  version          = "v1.6.1"
  namespace        = "cert-manager"
  create_namespace = true

  set {
    name  = "installCRDs"
    value = "true"
  }
}

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

# RunnerSet
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: {{ include "arc-runnerset.fullname" . }}
spec:
  ephemeral: true
  replicas: {{ .Values.runners.count }}
  repository: {{ .Values.github.repository }}
  dockerdWithinRunnerContainer: true
  image: {{ .Values.runner.image }}
  workDir: /home/runner/work
  dockerMTU: 1450
  labels:
    - k8s
    - dind
  selector:
    matchLabels:
      app: {{ include "arc-runnerset.fullname" . }}
  serviceName: {{ include "arc-runnerset.fullname" . }}
  template:
    metadata:
      labels:
        app: {{ include "arc-runnerset.fullname" . }}
    spec:
      containers:
        - name: runner
          env: []
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: docker
              mountPath: /var/lib/docker
            - name: gradle
              mountPath: /home/runner/.gradle
          resources:
            limits:
              memory: "5Gi"
            requests:
              memory: "2Gi"
  volumeClaimTemplates:
    - metadata:
        name: docker
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: local-path
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: gradle
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: local-path
        resources:
          requests:
            storage: 5Gi

To Reproduce

1. Configure volumeClaimTemplates for the RunnerSet
2. Run the workflows multiple times
3. Unproportional PVs are created from the workflow runs

Describe the bug

I configured volumeClaimTemplates for the RunnerSet and number of replicas is 5. The volumeClaimTemplates contains 2 persistent volume mappings - one for docker and another for gradle. The runners are configured as ephemeral: true.

At the start of the RunnerSet deployment 10 PVs (5*2 - one for docker and another for gradle) are created and bound to all the runner pods. When a newly assigned workflow is run and completed on a runner, the runner pod is deleted and a brand-new pod is created and listens for jobs. Unfortunately, the newly created pod does not attach the recently freed available PVs (from the deleted runner pod), but instead it creates new set of PVs and attaches them to it.

Over the period of time these redundant PVs accumulate and the system becomes out of disk space.

Describe the expected behavior

PVs created from the RunnerSet deployment should be used efficiently.

When a newly assigned workflow is run and completed on a runner, the newly created pod should attach the recently freed available PVs from the deleted runner pod.

Whole Controller Logs

2023-02-12T17:53:49Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2023-02-12T17:53:49Z	INFO	Initializing actions-runner-controller	{"version": "v0.27.0", "default-scale-down-delay": "10m0s", "sync-period": "1m0s", "default-runner-image": "summerwind/actions-runner:latest", "default-docker-image": "docker:dind", "common-runnner-labels": null, "leader-election-enabled": true, "leader-election-id": "actions-runner-controller", "watch-namespace": ""}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-02-12T17:53:49Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-runner-set-pod"}
2023-02-12T17:53:49Z	INFO	starting manager
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook.webhooks	Starting webhook server
2023-02-12T17:53:49Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2023-02-12T17:53:49Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
2023-02-12T17:53:49Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-02-12T17:53:49Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
I0212 17:53:49.873644       1 leaderelection.go:248] attempting to acquire leader lease actions-runner-system/actions-runner-controller...
I0212 17:54:06.978268       1 leaderelection.go:258] successfully acquired lease actions-runner-system/actions-runner-controller
2023-02-12T17:54:06Z	DEBUG	events	actions-runner-controller-86cb9958c6-jqm5h_576e790b-bfab-4da6-b8f6-8417f1a7ad26 became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"actions-runner-system","name":"actions-runner-controller","uid":"1a603795-f559-4b7e-8db4-2a41873f879f","apiVersion":"coordination.k8s.io/v1","resourceVersion":"23530004"}, "reason": "LeaderElection"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1alpha1.Runner"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1.Pod"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerDeployment"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.Runner"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1alpha1.RunnerSet"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1.StatefulSet"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "source": "kind source: *v1.PersistentVolume"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "source": "kind source: *v1alpha1.HorizontalRunnerAutoscaler"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler"}
2023-02-12T17:54:06Z	INFO	Starting EventSource	{"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-02-12T17:54:06Z	INFO	Starting Controller	{"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "worker count": 1}
2023-02-12T17:54:07Z	INFO	Starting workers	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "worker count": 1}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-xwdsl-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-v77jt-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-xwdsl-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-44lzs-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-h7fch-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-twzn2-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-fztg4-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-r6r7b-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-497t5-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-bgjjc-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-ftn9q-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-nndjv-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-agent-arc-runnerset-v77jt-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-h7fch-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-r6r7b-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-497t5-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-bgjjc-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-twzn2-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-fztg4-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-nndjv-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-44lzs-0", "requeueAfter": "10s"}
2023-02-12T17:54:07Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-ftn9q-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-xwdsl-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-v77jt-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-xwdsl-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-44lzs-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-h7fch-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-twzn2-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-fztg4-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-r6r7b-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-497t5-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-bgjjc-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-ftn9q-0", "requeueAfter": "10s"}
2023-02-12T17:54:17Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-nndjv-0", "requeueAfter": "10s"}
.
.
.
2023-02-13T15:30:00Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-qtszh-0", "requeueAfter": "10s"}
2023-02-13T15:30:00Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-qtszh-0", "requeueAfter": "10s"}
2023-02-13T15:30:01Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-nb6m5-0", "requeueAfter": "10s"}
2023-02-13T15:30:01Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-nb6m5-0", "requeueAfter": "10s"}
2023-02-13T15:30:02Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-jr8pg-0", "requeueAfter": "10s"}
2023-02-13T15:30:02Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-jr8pg-0", "requeueAfter": "10s"}
2023-02-13T15:30:03Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-zx5fr-0", "requeueAfter": "10s"}
2023-02-13T15:30:03Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-zx5fr-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-v77jt-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-agent-arc-runnerset-tb44r-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-agent-arc-runnerset-v77jt-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-x2r5j-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/gradle-vm-xxxx-arc-runnerset-x2r5j-0", "requeueAfter": "10s"}
2023-02-13T15:30:05Z	DEBUG	runnerpersistentvolumeclaim	Retrying sync until statefulset gets removed	{"pvc": "default/docker-vm-xxxx-arc-runnerset-7x925-0", "requeueAfter": "10s"}

Whole Runner Pod Logs

NA

Additional Context

NA

Start with the RunnerSet controller and the runnerpersistentvolume-controller mentioned in the logs, then reproduce the issue using the provided volumeClaimTemplates and ephemeral RunnerSet configuration. Trace what happens to PersistentVolumes after a runner pod is deleted; done means replacement pods reuse the freed volumes instead of creating an accumulating set of new PVs.