Pods stuck in terminating on AKS cluster with workload identity

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.26.0

Helm Chart Version

0.21.1

CertManager Version

Using AGIC + key vault cert (no issue)

Deployment Method

ArgoCD

cert-manager installation

Issue is not related to certificates. My webhook server works perfectly fine.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deployment-amd64-14gb
spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
    spec:
      organization: LYB-Digital
      labels:
        - linux
        - amd64
        - 14gb
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/github-runner-amd64
          operator: Exists
        - effect: NoSchedule
          key: kubernetes.azure.com/scalesetpriority
          value: spot
          operator: Equal
      ephemeral: true
      dockerdWithinRunnerContainer: true
      resources:
        limits:
          cpu: 3800m
          memory: 14Gi
        requests:
          cpu: 3800m
          memory: 14Gi
---
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-scaler-amd64-14gb
spec:
  minReplicas: 0
  maxReplicas: 10
  scaleTargetRef:
    kind: RunnerDeployment
    name: runner-deployment-amd64-14gb
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}
      duration: '30m0s'

To Reproduce

1. Use an AKS cluster with workflow identity support enabled
2. Allow any job to queue and run (successful or not, makes no difference)
3. Job completes, the runner and related resources (e.g., SA, RB, etc) is removed, but the pod gets stuck in terminating state

Describe the bug

After jobs complete, pods are stuck in terminating state, and moreover, they are unable to be patched.

The bug, reported in both the ARC logs as well as when I try to manually patch the resources to remove finalizers, is:

Error from server: admission webhook "mutation.azure-workload-identity.io" denied the request: serviceaccounts "runner-deployment-amd64-14gb-wprcp-mw6nt" not found

It appears that removal of the service account renders the pod in an error state where it can't be interacted with at all.

Describe the expected behavior

Pods should be removed cleanly.

Whole Controller Logs

https://gist.github.com/james-trousdale-lyb/afcc5b15979c2151ea5aaef14d49f369#file-arc-logs-txt

Whole Runner Pod Logs

https://gist.github.com/james-trousdale-lyb/afcc5b15979c2151ea5aaef14d49f369#file-runner-logs-txt

Additional Context

Please note that I did ArgoCD install of inflated helm chart, not manifests directly, FWIW.

Also, I'm unsure if this bug is really with ARC or with the workload identity controller. I know that's a preview feature.

Start with the AKS reproduction steps and the ARC and runner logs linked in the issue, focusing on pod deletion and the mutation.azure-workload-identity.io admission error. Trace the interaction between ARC cleanup and the workload identity webhook; done means ephemeral runner pods and related resources are removed cleanly after jobs complete without leaving pods stuck in Terminating.