ARC with AKS workload identity not working

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.0

Helm Chart Version

0.25.2

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

Using AGIC + key vault cert (no issue)

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: datapipelines
spec:
  template:
    metadata:
      labels:
        app: datapipelines
           azure.workload.identity/use: "true"
        annotations:
            azure.workload.identity/inject-proxy-sidecar: "true"
    spec:
      organization: organization
      image: 
      imagePullPolicy: Always
      serviceAccountName: datapipelines
      labels:
         - self-hosted
      ephemeral: true

To Reproduce

1. Use an AKS cluster with workflow identity support enabled
2. Allow any job to queue and run (successful or not, makes no difference)
3. Job completes, the runner and related resources are not scaling down

Describe the bug

Pods are not scaling down with AKS workload identity

Describe the expected behavior

Pod Sclaes down after grace period

Whole Controller Logs

2023-01-23T12:07:57Z ERROR runnerreplicaset Failed to patch pod to have actions-runner/unregistration-request-timestamp annotation {"runnerreplicaset": "gitrunners/adfdatapipelines-simv2-runners-t5pb2", "lastSyncTime": "2023-01-23T11:50:56Z", "effectiveTime": "<nil>", "templateHashDesired": "6d74d7fd7b", "replicasDesired": 0, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": ["6d74d7fd7b"], "owner": "gitrunners/adfdatapipelines-simv2-runners-t5pb2-zwl9b", "error": "Pod \"adfdatapipelines-simv2-runners-t5pb2-zwl9b\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.annotatePodOnce
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_graceful_stop.go:62
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.syncRunnerPodsOwners
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_owner.go:440
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReplicaSetReconciler).Reconcile
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runnerreplicaset_controller.go:131
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-01-23T12:07:57Z ERROR Reconciler error {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"adfdatapipelines-simv2-runners-t5pb2","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "adfdatapipelines-simv2-runners-t5pb2", "reconcileID": "7963ab72-50be-4ced-b195-62136ec426ba", "error": "Pod \"adfdatapipelines-simv2-runners-t5pb2-zwl9b\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-01-23T12:07:57Z INFO runnerreplicaset Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions/actions-runner-controller/issues/288 {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "owner": "gitrunners/infradeployment-simv2-runners-9psmm-7m5l9", "creationTimestamp": "2023-01-23 11:50:19 +0000 UTC", "readyTransitionTime": "2023-01-23 11:50:24 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-01-23T12:07:57Z INFO runnerreplicaset Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions/actions-runner-controller/issues/288 {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "owner": "gitrunners/infradeployment-simv2-runners-9psmm-sdbj2", "creationTimestamp": "2023-01-23 11:49:31 +0000 UTC", "readyTransitionTime": "2023-01-23 11:49:35 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-01-23T12:07:58Z ERROR runnerreplicaset Failed to patch pod to have actions-runner/unregistration-request-timestamp annotation {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "lastSyncTime": "2023-01-23T11:50:19Z", "effectiveTime": "<nil>", "templateHashDesired": "6dcdfbfd65", "replicasDesired": 0, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": ["6dcdfbfd65"], "owner": "gitrunners/infradeployment-simv2-runners-9psmm-7m5l9", "error": "Pod \"infradeployment-simv2-runners-9psmm-7m5l9\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.annotatePodOnce
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_graceful_stop.go:62
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.syncRunnerPodsOwners
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_owner.go:440
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReplicaSetReconciler).Reconcile
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runnerreplicaset_controller.go:131
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR Reconciler error {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"infradeployment-simv2-runners-9psmm","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "infradeployment-simv2-runners-9psmm", "reconcileID": "377548e7-5cc3-4c10-8bc5-b02a931bd7de", "error": "Pod \"infradeployment-simv2-runners-9psmm-7m5l9\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR runnerpod Failed to update runner {"runnerpod": "gitrunners/adfdatapipelines-simv2-runners-rwtph-zmv7g", "error": "Pod \"adfdatapipelines-simv2-runners-rwtph-zmv7g\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerPodReconciler).Reconcile
 github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
 sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"adfdatapipelines-simv2-runners-rwtph-zmv7g","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "adfdatapipelines-simv2-runners-rwtph-zmv7g", "reconcileID": "6edc5ae1-dca9-4b2c-a05d-f29abaa0f79f", "error": "Pod \"adfdatapipelines-simv2-runners-rwtph-zmv7g\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv:       nil,\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n  \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 15 identical elements\n  \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n  \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:  \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n  \t\t\t},\n  \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n  \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName:      \"azure-identity-token\",\n- \t\t\t\t\tReadOnly:  true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Never\",\n  \t... // 26 identical fields\n  }\n"}

Whole Runner Pod Logs

2023-01-23 12:02:28.891  NOTICE --- Runner init started with pid 10
2023-01-23 12:02:29.845  DEBUG --- Configuring the runner.
# Authentication
√ Connected to GitHub
# Runner Registration
√ Runner successfully added
√ Runner connection is good
# Runner settings
√ Settings Saved.
2023-01-23 12:02:37.820  DEBUG --- Runner successfully configured.
 {
  "isHostedServer": false,
  "agentId": 31827,
  "agentName": "datapipelines",
  "poolId": 9,
  "poolName": "datapipelines",
  "serverUrl": "https://github.com/_services/pipelines/tg1bKSIkI103oyxiraiSvb1IhsYTwVsA6Qhr1DxAimauUGR9mk",
  "gitHubUrl": "https://github.com/org",
  "workFolder": "/runner/_work"
2023-01-23 12:02:37.828  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2023-01-23 12:02:37.830  DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
}CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
√ Connected to GitHub
Current runner version: '2.299.1'
2023-01-23 12:02:46Z: Listening for Jobs

Additional Context

No response

Start with annotatePodOnce in controllers/actions.summerwind.net/runner_graceful_stop.go:62 and the reconciliation path in runner_pod_owner.go:440. Compare the pod mutation with the Azure workload identity-injected fields shown in the logs, then verify that runner unregistration and scale-down complete without a forbidden pod update after the grace period.