Runner failed to register itself to GitHub in timely manner

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.26.0

Helm Chart Version

0.21.1

CertManager Version

v1.9.1

Deployment Method

Helm

cert-manager installation

cert manager was installed with the repo: https://charts.jetstack.io a simple helm install.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-runners
  namespace: actions-runner-system
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: github-runners
    meta.helm.sh/release-namespace: actions-runner-system
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/name: runner-standard
    spec:
      organization: Gorillas-App
      labels:
        - runner-standard
        - self-hosted
      resources:
        requests:
          cpu: "1.9"
          memory: "6000Mi"
        limits:
          cpu: "3.8"
          memory: "14000Mi"
      image: summerwind/actions-runner-dind:v2.279.0-ubuntu-20.04-743e6d6
      dockerdWithinRunnerContainer: true
      serviceAccountName: ci-github-runners
      # Below volumeMount is WA for stuck docker/setup-buildx-action@v1 action in Hub workflow
      # Issue: https://github.com/docker/for-linux/issues/1015#issuecomment-841915668
      volumes:
        - name: opt-buildx
          emptyDir: { }
      volumeMounts:
        - mountPath: /opt/hostedtoolcache
          name: opt-buildx
      nodeSelector:
        cloud.google.com/gke-nodepool: ci-runners-default-node-pool
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: hra-github-runners
  namespace: actions-runner-system
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: github-runners
    meta.helm.sh/release-namespace: actions-runner-system
spec:
  scaleTargetRef:
    name: github-runners
  minReplicas: 5
  maxReplicas: 40
  scaleUpTriggers:
    scaleUpTriggers:
    - duration: 60m
      githubEvent:
        workflowJob: {}

To Reproduce

The runners are not scaling down or are stuck in the Terminating state.  The runners have the logs: 

Starting Runner listener with startup type: service
Started listener process
An error occurred: Not configured. Run config.(sh/cmd) to configure the runner.
Runner listener exited with error code 2

Describe the bug

The runners were running as expected until the actions-runner-controller was upgraded from 0.13.0 to 0.26.0. The CRDs were already upgraded along with the upgrade of the ARC.

The runners are being scaled up unnecessarily as the ARC is trying to fix the runners by recreating the pod when it is unable to register to Github. I have already checked the issue mentioned in the log message which is not applicable here.

Describe the expected behavior

The runner pods shoudl scale down when there are no jobs. Currently i have around 60 runners of which many are on the error state and deletion if these pods are only possible with removing the finaluzer on them manually.

Whole Controller Logs

2023-03-02T12:25:28Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-runner-set-pod", "code": 200, "reason": "", "UID": "76307700-b519-4d4b-92e2-02bca0f01bab", "allowed": true}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runner	Created runner pod	{"runner": "actions-runner-system/github-runners-large-6gt7v-29hvf", "repository": ""}
2023-03-02T12:25:28Z	DEBUG	events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"github-runners-large-6gt7v-29hvf","uid":"4c08be2c-8ba0-46e0-bac8-0ee7259a274c","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"481725020"}, "reason": "PodCreated", "message": "Created pod 'github-runners-large-6gt7v-29hvf'"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-fw7c2", "creationTimestamp": "2023-03-02 12:04:57 +0000 UTC", "readyTransitionTime": "2023-03-02 12:05:36 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-2ldx2", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:13:17 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-jb8j5", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:11:40 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-zkcrm", "creationTimestamp": "2023-03-02 12:14:42 +0000 UTC", "readyTransitionTime": "2023-03-02 12:14:44 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-nnhdj", "creationTimestamp": "2023-03-02 11:25:59 +0000 UTC", "readyTransitionTime": "2023-03-02 11:26:01 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-29hvf", "pods": [{"kind":"Pod","apiVersion":"v1","metadata":{"name":"github-runners-large-6gt7v-29hvf","namespace":"actions-runner-system","uid":"cae57f37-4238-4296-9c30-a3be61ec0ebc","resourceVersion":"481725025","creationTimestamp":"2023-03-02T12:25:28Z","labels":{"actions-runner":"","actions-runner-controller/inject-registration-token":"true","app.kubernetes.io/name":"runner-large","pod-template-hash":"5bb9df99b6","runner-deployment-name":"github-runners-large","runner-template-hash":"65669b56b5"},"annotations":{"actions-runner-controller/token-expires-at":"2023-03-02T14:25:28+01:00","sync-time":"2023-03-02T12:25:28Z"},"ownerReferences":[{"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"Runner","name":"github-runners-large-6gt7v-29hvf","uid":"4c08be2c-8ba0-46e0-bac8-0ee7259a274c","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"v1","time":"2023-03-02T12:25:28Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:sync-time":{}},"f:labels":{".":{},"f:actions-runner":{},"f:actions-runner-controller/inject-registration-token":{},"f:app.kubernetes.io/name":{},"f:pod-template-hash":{},"f:runner-deployment-name":{},"f:runner-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"4c08be2c-8ba0-46e0-bac8-0ee7259a274c\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"runner\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"DOCKERD_IN_RUNNER\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DOCKER_ENABLED\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"GITHUB_URL\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_ENTERPRISE\"}":{".":{},"f:name":{}},"k:{\"name\":\"RUNNER_EPHEMERAL\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_GROUP\"}":{".":{},"f:name":{}},"k:{\"name\":\"RUNNER_LABELS\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_ORG\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_REPO\"}":{".":{},"f:name":{}},"k:{\"name\":\"RUNNER_STATUS_UPDATE_HOOK\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_TOKEN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"RUNNER_WORKDIR\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:privileged":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/opt/hostedtoolcache\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/runner\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/var/run/datadog\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:nodeSelector":{".":{},"f:cloud.google.com/gke-nodepool":{}},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"dsdsocket\"}":{".":{},"f:hostPath":{".":{},"f:path":{},"f:type":{}},"f:name":{}},"k:{\"name\":\"opt-buildx\"}":{".":{},"f:emptyDir":{},"f:name":{}},"k:{\"name\":\"runner\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}}]},"spec":{"volumes":[{"name":"runner","emptyDir":{}},{"name":"dsdsocket","hostPath":{"path":"/var/run/datadog/","type":""}},{"name":"opt-buildx","emptyDir":{}}],"containers":[{"name":"runner","image":"summerwind/actions-runner-dind:v2.279.0-ubuntu-20.04-743e6d6","env":[{"name":"RUNNER_ORG","value":"Gorillas-App"},{"name":"RUNNER_REPO"},{"name":"RUNNER_ENTERPRISE"},{"name":"RUNNER_LABELS","value":"runner-large,self-hosted"},{"name":"RUNNER_GROUP"},{"name":"DOCKER_ENABLED","value":"true"},{"name":"DOCKERD_IN_RUNNER","value":"true"},{"name":"GITHUB_URL","value":"https://github.com/"},{"name":"RUNNER_WORKDIR","value":"/runner/_work"},{"name":"RUNNER_EPHEMERAL","value":"true"},{"name":"RUNNER_STATUS_UPDATE_HOOK","value":"false"},{"name":"RUNNER_NAME","value":"github-runners-large-6gt7v-29hvf"},{"name":"RUNNER_TOKEN","value":"AUHJVIYKZFHCNTGLUSUHWPDEACR4Q"}],"resources":{"limits":{"cpu":"7500m","memory":"30000Mi"},"requests":{"cpu":"1900m","memory":"6000Mi"}},"volumeMounts":[{"name":"runner","mountPath":"/runner"},{"name":"dsdsocket","mountPath":"/var/run/datadog"},{"name":"opt-buildx","mountPath":"/opt/hostedtoolcache"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"privileged":true}}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"cloud.google.com/gke-nodepool":"ci-runners-default-node-pool"},"serviceAccountName":"ci-github-runners","serviceAccount":"ci-github-runners","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","qosClass":"Burstable"}}]}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-fw7c2", "creationTimestamp": "2023-03-02 12:04:57 +0000 UTC", "readyTransitionTime": "2023-03-02 12:05:36 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-2ldx2", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:13:17 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-jb8j5", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:11:40 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-zkcrm", "creationTimestamp": "2023-03-02 12:14:42 +0000 UTC", "readyTransitionTime": "2023-03-02 12:14:44 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-nnhdj", "creationTimestamp": "2023-03-02 11:25:59 +0000 UTC", "readyTransitionTime": "2023-03-02 11:26:01 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-9j9xp", "creationTimestamp": "2023-03-02 11:42:19 +0000 UTC", "readyTransitionTime": "2023-03-02 11:42:21 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-rmwvx", "creationTimestamp": "2023-03-02 11:52:55 +0000 UTC", "readyTransitionTime": "2023-03-02 12:00:26 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-2ldx2", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:13:17 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-jb8j5", "creationTimestamp": "2023-03-02 12:11:34 +0000 UTC", "readyTransitionTime": "2023-03-02 12:11:40 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-zkcrm", "creationTimestamp": "2023-03-02 12:14:42 +0000 UTC", "readyTransitionTime": "2023-03-02 12:14:44 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-nnhdj", "creationTimestamp": "2023-03-02 11:25:59 +0000 UTC", "readyTransitionTime": "2023-03-02 11:26:01 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-9j9xp", "creationTimestamp": "2023-03-02 11:42:19 +0000 UTC", "readyTransitionTime": "2023-03-02 11:42:21 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-03-02T12:25:28Z	INFO	actions-runner-controller.runnerreplicaset	Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions-runner-controller/actions-runner-controller/issues/288	{"runnerreplicaset": "actions-runner-system/github-runners-large-6gt7v", "owner": "actions-runner-system/github-runners-large-6gt7v-rmwvx", "creationTimestamp": "2023-03-02 11:52:55 +0000 UTC", "readyTransitionTime": "2023-03-02 12:00:26 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}

Whole Runner Pod Logs

Starting Runner listener with startup type: service
Started listener process
An error occurred: Not configured. Run config.(sh/cmd) to configure the runner.
Runner listener exited with error code 2
Runner listener exit with retryable error, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
An error occurred: Not configured. Run config.(sh/cmd) to configure the runner.
Runner listener exited with error code 2
Runner listener exit with retryable error, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process

Additional Context

I have already tried restarting the ARC pod which has not made anything better.

Start with the RunnerReplicaSet controller behavior described in the logs and compare the 0.13.0 and 0.26.0 behavior around the registration timeout. Reproduce the Runner listener's “Not configured” state using the supplied RunnerDeployment and HorizontalRunnerAutoscaler definitions; done means runners register successfully and idle runners scale down without stuck Terminating pods.