possible race condition in helm chart v0.14.2 for listener setup

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.2

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

use attached ansible playbooks and values yaml files to move arc runner scalesets from one AKS to another.
Every now and then the helm upgrade -install leads to the listener going into a crash boot loop.

Describe the bug

I am trying to shift loads of actions runner scalesets from one AKS to another using ansible.
I have attached the two yamls I am using for the execution flow (shift_prod_to_test_no_ids.yaml and shift_single_scaleset.yaml) along with the supporting values.yml files used (runner-values-dind-rootless.yaml and values-win.yaml).
What I am experiencing is intermittent. Sometimes the shift process works fine, every 2nd or 3rd time however, the new listener installation of the cluster where the arc runner set is being moved TO is going into an endless crash boot loop with error messages like the following:

time=2026-09-14T09:03:01.282Z level=INFO source=github.com/actions/scaleset@v0.4.0/client.go:988 msg="refreshing token" githubConfigUrl=https://ext-github.swm.de/SWM
time=2026-09-14T09:03:01.284Z level=INFO source=github.com/actions/scaleset@v0.4.0/client.go:806 msg="getting access token for GitHub App auth" accessTokenURL=https://ext-github.swm.de/api/v3/app/installations/4/access_tokens
time=2026-09-14T09:03:01.284Z level=DEBUG source=github.com/hashicorp/go-retryablehttp@v0.7.8/client.go:678 msg="performing request" method=POST url=https://ext-github.swm.de/api/v3/app/installations/4/access_tokens
time=2026-09-14T09:03:01.399Z level=INFO source=github.com/actions/scaleset@v0.4.0/client.go:765 msg="getting runner registration token" registrationTokenURL=https://ext-github.swm.de/api/v3/orgs/SWM/actions/runners/registration-token
time=2026-09-14T09:03:01.399Z level=DEBUG source=github.com/hashicorp/go-retryablehttp@v0.7.8/client.go:678 msg="performing request" method=POST url=https://ext-github.swm.de/api/v3/orgs/SWM/actions/runners/registration-token
time=2026-09-14T09:03:01.460Z level=INFO source=github.com/actions/scaleset@v0.4.0/client.go:858 msg="getting Actions tenant URL and JWT" registrationURL=https://ext-github.swm.de/api/v3/actions/runner-registration
time=2026-09-14T09:03:01.460Z level=DEBUG source=github.com/hashicorp/go-retryablehttp@v0.7.8/client.go:678 msg="performing request" method=POST url=https://ext-github.swm.de/api/v3/actions/runner-registration
time=2026-09-14T09:03:01.591Z level=DEBUG source=github.com/hashicorp/go-retryablehttp@v0.7.8/client.go:678 msg="performing request" method=POST url="https://ext-github.swm.de/_services/pipelines/5I2uObSEULKpu2zKXgnz5ja8WseIrz8rcjLIYShYwjrochGeas/_apis/runtime/runnerscalesets/99/sessions?api-version=6.0-preview"
time=2026-09-14T09:03:01.691Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/main.go:134 msg="Starting listener"
time=2026-09-14T09:03:01.691Z level=INFO source=github.com/actions/scaleset@v0.4.0/listener/listener.go:160 msg="Handling initial session statistics" component=listener totalAssignedJobs=0
time=2026-09-14T09:03:01.691Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:250 msg="Calculated target runner count" component=worker "assigned job"=0 decision=1 min=1 max=200 currentRunnerCount=1
time=2026-09-14T09:03:01.693Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:197 msg=Compare component=worker original="{"metadata":{},"spec":{"replicas":-1,"patchID":-1,"ephemeralRunnerSpec":{"metadata":{},"spec":{"containers":null}}},"status":{"currentReplicas":0,"pendingEphemeralRunners":0,"runningEphemeralRunners":0,"failedEphemeralRunners":0,"phase":""}}" patch="{"metadata":{},"spec":{"replicas":1,"patchID":0,"ephemeralRunnerSpec":{"metadata":{},"spec":{"containers":null}}},"status":{"currentReplicas":0,"pendingEphemeralRunners":0,"runningEphemeralRunners":0,"failedEphemeralRunners":0,"phase":""}}"
time=2026-09-14T09:03:01.693Z level=INFO source=github.com/actions/actions-runner-controller/cmd/ghalistener/scaler/scaler.go:203 msg="Preparing EphemeralRunnerSet update" component=worker json="{"spec":{"patchID":0,"replicas":1}}"
time=2026-09-14T09:03:01.726Z level=DEBUG source=github.com/hashicorp/go-retryablehttp@v0.7.8/client.go:678 msg="performing request" method=DELETE url="https://ext-github.swm.de/_services/pipelines/5I2uObSEULKpu2zKXgnz5ja8WseIrz8rcjLIYShYwjrochGeas/_apis/runtime/runnerscalesets/99/sessions/fd828755-3683-41bb-9f49-851ba45fa90e?api-version=6.0-preview"
2026/09/14 09:03:01 Application returned an error: handling initial message failed: could not patch ephemeral runner set , patch JSON: {"spec":{"patchID":0,"replicas":1}}, error: ephemeralrunnersets.actions.github.com "arc-runner-set-swm-win-8wrsc" not found

The interesting thing about it is, that the ephemeralrunnerset mentioned in the error message does not exist in the respective namespace but instead an ephemeralrunnerset with a different name is there:

(base) ➜ ansible_quickstart kubectl -n arc-swm get ephemeralrunnersets
NAME DESIREDREPLICAS CURRENTREPLICAS PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS
arc-runner-set-swm-hspmg 1 1 0 1
arc-runner-set-swm-win-h89vv 0
dependabot-w4w9b

When I patch the crashing listener to reference the correct ephemeralrunnerset like this:

(base) ➜ ansible_quickstart kubectl -n arc patch AutoscalingListener/arc-runner-set-swm-win-54546898-listener -p '{"spec":{"ephemeralRunnerSetName":"arc-runner-set-swm-win-h89vv"}}' --type=merge
autoscalinglistener.actions.github.com/arc-runner-set-swm-win-54546898-listener patched

the execution of the ansible playbook continues just fine.

I believe that the listener installation in the helm chart is suffering from some kind of race condition which causes the correct name of the ephemeralrunnerset in the listener to sometimes be incorrect.

When I do the same process of shifting the runnerscaleset from one cluster to another purely from command line by cutting and pasting the respective commands, it always works flawlessly.

Describe the expected behavior

shift_prod_to_test_no_ids.yaml
shift_single_scaleset.yaml

runner-values-dind-rootless.yaml
values-win.yaml

Additional Context

## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: ""

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
githubConfigSecret:
  ### GitHub Apps Configuration
  ## NOTE: IDs MUST be strings, use quotes
  #github_app_id: ""
  #github_app_installation_id: ""
  #github_app_private_key: |

  ### GitHub PAT Configuration
  github_token: ""
## If you have a pre-define Kubernetes secret in the same namespace the gha-runner-scale-set is going to deploy,
## you can also reference it via `githubConfigSecret: pre-defined-secret`.
## You need to make sure your predefined secret has all the required secret data set properly.
##   For a pre-defined secret using GitHub PAT, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_token='ghp_your_pat'
##   For a pre-defined secret using GitHub App, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_app_id=123456 --from-literal=github_app_installation_id=654321 --from-literal=github_app_private_key='-----BEGIN CERTIFICATE-----*******'
# githubConfigSecret: pre-defined-secret

## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
# proxy:
#   http:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   https:
#     url: http://proxy.com:1234
#     credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
#   noProxy:
#     - example.com
#     - example.org

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 200

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 1

# runnerGroup: "default"

## name of the runner scale set to create.  Defaults to the helm release name
# runnerScaleSetName: ""

## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
##   specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
##   2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
# githubServerTLS:
#   certificateFrom:
#     configMapKeyRef:
#       name: config-map-name
#       key: ca.crt
#   runnerMountPath: /usr/local/share/ca-certificates/

## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
# containerMode:
#   type: "dind"  ## type can be set to dind or kubernetes
#   ## the following is required when containerMode.type=kubernetes
#   kubernetesModeWorkVolumeClaim:
#     accessModes: ["ReadWriteOnce"]
#     # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
#     storageClassName: "dynamic-blob-storage"
#     resources:
#       requests:
#         storage: 1Gi
#   kubernetesModeServiceAccount:
#     annotations:

## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000
#     # Use this section to add the configuration of a side-car container.
#     # Comment it out or remove it if you don't need it.
#     # Spec for this container will be applied as is without any modifications.
#     - name: side-car
#       image: example-sidecar
listenerTemplate:
  # listeners should not be evictable due to stability reasons and metrics contiuity
#  metadata:
#    annotations:
#      cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
  spec:
    containers:
      - name: listener
        image: ghcr.io/actions/gha-runner-scale-set-controller:0.14.2
        securityContext:
          runAsUser: 1000

## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
  # metadata:
  #  annotations:
  #    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
  ## template.spec will be modified if you change the container mode
  ## with containerMode.type=dind, we will populate the template.spec with following pod spec
  ## template:
  spec:
    initContainers:
    - name: init-dind-externals
      # image: ghcr.io/actions/actions-runner:latest
      image: githubprodcr.azurecr.io/actions/actions-runner:2.337.0-swm
      command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
      volumeMounts:
        - name: dind-externals
          mountPath: /home/runner/tmpDir
    containers:
    - name: runner
      # image: ghcr.io/actions/actions-runner:latest
      image: githubprodcr.azurecr.io/actions/actions-runner:2.337.0-swm
      securityContext:
        runAsUser: 1001
        runAsGroup: 1001
        privileged: false
      resources:
        requests:
          cpu: 1
      #   limits:
      #     cpu: 2
      lifecycle:
        postStart:
          exec:
            command: [ "/bin/bash", "-c", "sudo ln -s /run/docker/docker.sock /var/run/docker.sock" ]
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: unix:///run/docker/docker.sock
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
          readOnly: true
    - name: dind
      image: githubprodcr.azurecr.io/docker:dind-rootless-swm-20250718
      securityContext:
        runAsUser: 1001
        runAsGroup: 1001
        privileged: true
      args:
        - dockerd
        - --host=unix:///run/user/1001/docker.sock
        - --dns=10.0.0.10
        # - --group=$(DOCKER_GROUP_GID)
        - --default-address-pool=base=172.30.0.0/16,size=24
      # env:
      #   - name: DOCKER_GROUP_GID
      #     value: "123"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/user/1001
        - name: dind-externals
          mountPath: /home/runner/externals
    volumes:
    - name: work
      emptyDir: {}
    - name: dind-sock
      emptyDir: {}
    - name: dind-externals
      emptyDir: {}
    imagePullSecrets:
    - name: regcred-acr
  ######################################################################################################
  ## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
  ## template:
  ##   spec:
  ##     containers:
  ##     - name: runner
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       command: ["/home/runner/run.sh"]
  ##       env:
  ##         - name: ACTIONS_RUNNER_CONTAINER_HOOKS
  ##           value: /home/runner/k8s/index.js
  ##         - name: ACTIONS_RUNNER_POD_NAME
  ##           valueFrom:
  ##             fieldRef:
  ##               fieldPath: metadata.name
  ##         - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
  ##           value: "true"
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##     volumes:
  ##       - name: work
  ##         ephemeral:
  ##           volumeClaimTemplate:
  ##             spec:
  ##               accessModes: [ "ReadWriteOnce" ]
  ##               storageClassName: "local-path"
  ##               resources:
  ##                 requests:
  ##                   storage: 1Gi
  # spec:
  #   containers:
  #     - name: runner
  #       image: ghcr.io/actions/actions-runner:latest
  #       command: ["/home/runner/run.sh"]

## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
#   namespace: arc-system
#   name: test-arc-gha-runner-scale-set-controller

Controller Logs

https://gist.github.com/alaric11/e1416c317a3046629a4296a069f8bf81

Runner Pod Logs

https://gist.github.com/alaric11/44c3330f477635b658b784482ffc80ae
https://gist.github.com/alaric11/6716a78621df75b1d13b631e3853824c
https://gist.github.com/alaric11/3677e6fee02c139839d4066f3cc7a244

Start by reproducing the Helm upgrade/install flow from the attached Ansible playbooks and values YAML files, then trace listener setup and the AutoscalingListener ephemeralRunnerSetName assignment. Compare the generated listener reference with the EphemeralRunnerSet created in the target namespace and verify repeated migrations no longer enter a crash loop or require manual patching.