chart: Limiting RABC permissions to the watch namespace when specified

Controller Version

0.22.0

Helm Chart Version

No response

Deployment Method

Other

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC to the latest and it didn't fix the issue

Resource Definitions

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: controller-manager
  namespace: actions-runner-system
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      serviceAccountName: actions-runner-system-admin
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --sync-period=10m
        command:
        - /manager
        env:
        - name: GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              key: github_token
              name: controller-manager
              optional: true
        - name: GITHUB_APP_ID
          valueFrom:
            secretKeyRef:
              key: github_app_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_INSTALLATION_ID
          valueFrom:
            secretKeyRef:
              key: github_app_installation_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_PRIVATE_KEY
          value: /etc/actions-runner-controller/github_app_private_key
        image: devtools-docker-local.decc-af.eng.vmware.com/wud/summerwind/actions-runner-controller:v0.22.0
        name: manager
        ports:
        - containerPort: 9443
          name: webhook-server
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 20Mi
        volumeMounts:
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
        - mountPath: /etc/actions-runner-controller
          name: controller-manager
          readOnly: true
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        - --watch-namespace actions-runner-system
        image: harbor-repo.vmware.com/dockerhub-proxy-cache/bitnami/kube-rbac-proxy:0.11.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
      terminationGracePeriodSeconds: 10
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: webhook-server-cert
      - name: controller-manager
        secret:
          secretName: controller-manager

To Reproduce

Refer https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.22.0/actions-runner-controller.yaml
1. deploy all the CustomResourceDefinition 
2. deploy namespace actions-runner-system
3. deploy Role, ClusterRole, RoleBinding in v0.22.0/actions-runner-controller.yaml, change the ClusterRoleBinding to RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: manager-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: actions-runner-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: proxy-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: proxy-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: actions-runner-system
4. deploy Deployment controller-manager with '--watch-namespace actions-runner-system'

Describe the bug

When --watch-namespace is set, controller still wants permissions on the cluster side.
Could controller performs on the namespace level with '-watch-namespace'?

Describe the expected behavior

controller performs on the namespace level with '-watch-namespace', seek for permission in namespaced instead of cluster scope

Controller Logs

E0513 19:38:31.034204       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1alpha1.RunnerReplicaSet: failed to list *v1alpha1.RunnerReplicaSet: runnerreplicasets.actions.summerwind.dev is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-system-admin" cannot list resource "runnerreplicasets" in API group "actions.summerwind.dev" at the cluster scope
W0513 19:39:00.209867       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-system-admin" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0513 19:39:00.209913       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-system-admin" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0513 19:39:21.288471       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1alpha1.RunnerReplicaSet: runnerreplicasets.actions.summerwind.dev is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-system-admin" cannot list resource "runnerreplicasets" in API group "actions.summerwind.dev" at the cluster scope

Runner Pod Logs

None

Additional Context

No response

Start with the v0.22.0 actions-runner-controller.yaml manifest and the Deployment's --watch-namespace configuration, then inspect the ClusterRole, RoleBinding, and controller logs described here. Reproduce the deployment using namespaced RoleBindings and verify whether the controller still issues cluster-scope list requests. Done means the configured namespace can be watched without the reported cluster-scope permission errors.