Runner pods stuck terminating for containerMode: kubernetes and watchNamespace

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

not installed

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: v1
kind: Namespace
metadata:
  name: runner-namespace

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: runner-namespace
  name: runner-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["get", "create"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "watch",]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "create", "delete"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: runner-service-account
  namespace: runner-namespace

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: runner-role-binding
  namespace: runner-namespace
subjects:
- kind: ServiceAccount
  name: runner-service-account
  apiGroup: ""
roleRef:
  kind: Role
  name: runner-role
  apiGroup: ""

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
  namespace: runner-namespace
spec:
  replicas: 1
  template:
    spec:
      containerMode: kubernetes
      serviceAccountName: runner-service-account

To Reproduce

1. Deploy ARC in one namespace with `scope.singleNamespace: true` and `scope.watchNamespace: runner-namespace`
2. Deploy runner in another namespace `runner-namespace` with `containerMode: kubernetes` (see above)
3. Run a workflow

Describe the bug

Runner Pod is stuck terminating due to finalizers, ARC logs:
2023-06-05T12:00:53Z INFO runnerpod Runner-linked secrets clean up that has failed due to an error. If this persists, please manually remove the runner-linked secrets to unblock ARC {"runnerpod": "runner-namespace/example-runnerdeploy-6rhhk-wvn5s", "err": "failed to list runner-linked secrets: secrets is forbidden: User \"system:serviceaccount:actions-runner-system:actions-runner-controller\" cannot list resource \"secrets\" in API group \"\" in the namespace \"runner-namespace\""}

Describe the expected behavior

Runner Pod terminates properly

Whole Controller Logs

https://gist.github.com/AlexLukasT/d1dd4de3492cb294d5d61a3d8363c9c6#file-controller-log

Whole Runner Pod Logs

https://gist.github.com/AlexLukasT/d1dd4de3492cb294d5d61a3d8363c9c6#file-runner-log

Additional Context

When not using containerMode: kubernetes or not limiting the ARC to one namespace it is working fine, so I suspect the issues occurs only when both are used.

Start by reproducing the issue with scope.singleNamespace, watchNamespace, and containerMode: kubernetes using the resource definitions in the report. Inspect the controller logs around runner-linked secret cleanup and the namespace-scoped RBAC rules. Done means the runner pod terminates without a forbidden secrets error and cleanup succeeds.