Unable to Register an Org runner with Github App creds

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.22.0

Helm Chart Version

0.22.0

CertManager Version

No response

Deployment Method

ArgoCD

cert-manager installation

yes

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: org-github-self-hosted-runner
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      githubAPICredentialsFrom:
         secretRef:
           name: controller-manager-gp
      # https://github.com/actions/actions-runner-controller/blob/master/docs/choosing-runner-destination.md
      organization: unzercorp
      image: github/runner:0.0.9
      resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.8
              memory: 800Mi
      labels:
        - org-github-self-hosted-runner
      env:
        - name: RUNNER_ORG
          value: unzercorp
        - name: GITHUB_OWNER
          value: unzercorp
        - name: RUNNER_REPO
          value: ""
        - name: ORG_NAME
          value: "unzercorp"

----

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: org-github-self-hosted-runner-deployment-autoscaler
  namespace: actions-runner-system
spec:
  githubAPICredentialsFrom:
    secretRef:
      name: controller-manager-gp
  scaleDownDelaySecondsAfterScaleOut: 30
  scaleTargetRef:
    # Your RunnerDeployment Here
    name: org-github-self-hosted-runner
    kind: RunnerDeployment
  minReplicas: 1
  maxReplicas: 5
  
  metrics:
    # https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md
    - type: PercentageRunnersBusy
      scaleUpThreshold: '0.75'    # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
      scaleDownThreshold: '0.3'   # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
      scaleUpAdjustment: 1        # The scale up runner count added to desired count
      scaleDownAdjustment: 1      # The scale down runner count subtracted from the desired count

To Reproduce

just use the above config and create an org runner with github app credentials as per : https://github.com/actions/actions-runner-controller/blob/master/docs/authenticating-to-the-github-api.md

Describe the bug

Trying to create an org runner with Github App credentials using the yaml file above and runner keep recreating indefinitely and it unable to register.

Describe the expected behavior

Need to create an org runner with Github App credentials

Whole Controller Logs

2023-03-31T13:36:40Z	DEBUG	events	Created pod 'org-github-self-hosted-runner-5mmzk-zgwrf'	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-zgwrf","uid":"3bf6a074-4186-4221-9ba8-a477d1b9c31e","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589497"}, "reason": "PodCreated"}
2023-03-31T13:36:40Z	INFO	runnerpod	Unregistration started before runner ID is assigned and the runner stopped before obtaining ID within registration timeout. Perhaps the runner successfully ran the job and stopped normally before the runner ID becomes visible via GitHub API? Perhaps the runner pod was terminated by anyone other than ARC? Was it OOM killed? Marking unregistration as completed anyway because there's nothing ARC can do.	{"runnerpod": "actions-runner-system/org-github-self-hosted-runner-5mmzk-ckbcl", "registrationTimeout": "10m0s"}
2023-03-31T13:36:41Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-zgwrf", "podCreationTimestamp": "2023-03-31 13:36:40 +0000 UTC"}
2023-03-31T13:36:42Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-zgwrf", "podCreationTimestamp": "2023-03-31 13:36:40 +0000 UTC"}
2023-03-31T13:36:43Z	INFO	runner-resource	validate resource to be created	{"name": "org-github-self-hosted-runner-5mmzk-z529t"}
2023-03-31T13:36:43Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-zgwrf"}
2023-03-31T13:36:43Z	DEBUG	runnerreplicaset	Created replica(s)	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "544cf75c47", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-03-31T13:36:43Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "pods": null}
2023-03-31T13:36:43Z	INFO	runner	Removed finalizer	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-zgwrf"}
2023-03-31T13:36:43Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "pods": null}
2023-03-31T13:36:43Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-z529t"}
2023-03-31T13:36:43Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "pods": null}
2023-03-31T13:36:43Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "pods": null}
2023-03-31T13:36:43Z	INFO	runner	Updated registration token	{"runner": "org-github-self-hosted-runner-5mmzk-z529t", "repository": ""}
2023-03-31T13:36:43Z	DEBUG	events	Successfully update registration token	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-z529t","uid":"48d218df-7f57-4f38-af98-3b5ddbb7b1a9","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589559"}, "reason": "RegistrationTokenUpdated"}
2023-03-31T13:36:43Z	INFO	runner	Created runner pod	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "repository": ""}
2023-03-31T13:36:43Z	DEBUG	events	Created pod 'org-github-self-hosted-runner-5mmzk-z529t'	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-z529t","uid":"48d218df-7f57-4f38-af98-3b5ddbb7b1a9","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589561"}, "reason": "PodCreated"}
2023-03-31T13:36:43Z	INFO	runnerpod	Unregistration started before runner ID is assigned and the runner stopped before obtaining ID within registration timeout. Perhaps the runner successfully ran the job and stopped normally before the runner ID becomes visible via GitHub API? Perhaps the runner pod was terminated by anyone other than ARC? Was it OOM killed? Marking unregistration as completed anyway because there's nothing ARC can do.	{"runnerpod": "actions-runner-system/org-github-self-hosted-runner-5mmzk-zgwrf", "registrationTimeout": "10m0s"}
2023-03-31T13:36:45Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "podCreationTimestamp": "2023-03-31 13:36:43 +0000 UTC"}
2023-03-31T13:36:45Z	INFO	runner-resource	validate resource to be created	{"name": "org-github-self-hosted-runner-5mmzk-hlp7c"}
2023-03-31T13:36:45Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-z529t"}
2023-03-31T13:36:45Z	DEBUG	runnerreplicaset	Created replica(s)	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "544cf75c47", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-03-31T13:36:45Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "pods": null}
2023-03-31T13:36:45Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "pods": null}
2023-03-31T13:36:45Z	INFO	runner	Removed finalizer	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t"}
2023-03-31T13:36:45Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-hlp7c"}
2023-03-31T13:36:45Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "pods": null}
2023-03-31T13:36:45Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "pods": null}
2023-03-31T13:36:45Z	INFO	runner	Updated registration token	{"runner": "org-github-self-hosted-runner-5mmzk-hlp7c", "repository": ""}
2023-03-31T13:36:45Z	DEBUG	events	Successfully update registration token	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-hlp7c","uid":"72ea30e3-52e5-43b5-89f3-9fd3d896173f","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589610"}, "reason": "RegistrationTokenUpdated"}
2023-03-31T13:36:45Z	INFO	runner	Created runner pod	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "repository": ""}
2023-03-31T13:36:45Z	DEBUG	events	Created pod 'org-github-self-hosted-runner-5mmzk-hlp7c'	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-hlp7c","uid":"72ea30e3-52e5-43b5-89f3-9fd3d896173f","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589612"}, "reason": "PodCreated"}
2023-03-31T13:36:45Z	INFO	runnerpod	Unregistration started before runner ID is assigned and the runner stopped before obtaining ID within registration timeout. Perhaps the runner successfully ran the job and stopped normally before the runner ID becomes visible via GitHub API? Perhaps the runner pod was terminated by anyone other than ARC? Was it OOM killed? Marking unregistration as completed anyway because there's nothing ARC can do.	{"runnerpod": "actions-runner-system/org-github-self-hosted-runner-5mmzk-z529t", "registrationTimeout": "10m0s"}
2023-03-31T13:36:46Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "podCreationTimestamp": "2023-03-31 13:36:45 +0000 UTC"}
2023-03-31T13:36:47Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "podCreationTimestamp": "2023-03-31 13:36:45 +0000 UTC"}
2023-03-31T13:36:47Z	INFO	runner-resource	validate resource to be created	{"name": "org-github-self-hosted-runner-5mmzk-l9zk4"}
2023-03-31T13:36:47Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-hlp7c"}
2023-03-31T13:36:47Z	DEBUG	runnerreplicaset	Created replica(s)	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "544cf75c47", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-03-31T13:36:47Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "pods": null}
2023-03-31T13:36:47Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "pods": null}
2023-03-31T13:36:47Z	INFO	runner	Removed finalizer	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c"}
2023-03-31T13:36:47Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-l9zk4"}
2023-03-31T13:36:47Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "pods": null}
2023-03-31T13:36:47Z	INFO	runner	Updated registration token	{"runner": "org-github-self-hosted-runner-5mmzk-l9zk4", "repository": ""}
2023-03-31T13:36:47Z	DEBUG	events	Successfully update registration token	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-l9zk4","uid":"b7b1b0ba-0896-43cc-8c88-33c4a4f61535","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589673"}, "reason": "RegistrationTokenUpdated"}
2023-03-31T13:36:47Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "pods": null}
2023-03-31T13:36:47Z	INFO	runner	Created runner pod	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "repository": ""}
2023-03-31T13:36:47Z	DEBUG	events	Created pod 'org-github-self-hosted-runner-5mmzk-l9zk4'	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-l9zk4","uid":"b7b1b0ba-0896-43cc-8c88-33c4a4f61535","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589675"}, "reason": "PodCreated"}
2023-03-31T13:36:47Z	INFO	runnerpod	Unregistration started before runner ID is assigned and the runner stopped before obtaining ID within registration timeout. Perhaps the runner successfully ran the job and stopped normally before the runner ID becomes visible via GitHub API? Perhaps the runner pod was terminated by anyone other than ARC? Was it OOM killed? Marking unregistration as completed anyway because there's nothing ARC can do.	{"runnerpod": "actions-runner-system/org-github-self-hosted-runner-5mmzk-hlp7c", "registrationTimeout": "10m0s"}
2023-03-31T13:36:49Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "podCreationTimestamp": "2023-03-31 13:36:47 +0000 UTC"}
2023-03-31T13:36:50Z	DEBUG	runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "podCreationTimestamp": "2023-03-31 13:36:47 +0000 UTC"}
2023-03-31T13:36:50Z	INFO	runner-resource	validate resource to be created	{"name": "org-github-self-hosted-runner-5mmzk-8rb7f"}
2023-03-31T13:36:50Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-l9zk4"}
2023-03-31T13:36:50Z	DEBUG	runnerreplicaset	Created replica(s)	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "544cf75c47", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-03-31T13:36:50Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-8rb7f", "pods": null}
2023-03-31T13:36:50Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-8rb7f", "pods": null}
2023-03-31T13:36:50Z	INFO	runner	Removed finalizer	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4"}
2023-03-31T13:36:50Z	INFO	runner-resource	validate resource to be updated	{"name": "org-github-self-hosted-runner-5mmzk-8rb7f"}
2023-03-31T13:36:50Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-8rb7f", "pods": null}
2023-03-31T13:36:50Z	INFO	runner	Updated registration token	{"runner": "org-github-self-hosted-runner-5mmzk-8rb7f", "repository": ""}
2023-03-31T13:36:50Z	DEBUG	events	Successfully update registration token	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-8rb7f","uid":"301f0ac6-ca28-4aef-82f1-6aeef245cc43","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589734"}, "reason": "RegistrationTokenUpdated"}
2023-03-31T13:36:50Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/org-github-self-hosted-runner-5mmzk", "owner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-8rb7f", "pods": null}
2023-03-31T13:36:50Z	INFO	runner	Created runner pod	{"runner": "actions-runner-system/org-github-self-hosted-runner-5mmzk-8rb7f", "repository": ""}
2023-03-31T13:36:50Z	DEBUG	events	Created pod 'org-github-self-hosted-runner-5mmzk-8rb7f'	{"type": "Normal", "object": {"kind":"Runner","namespace":"actions-runner-system","name":"org-github-self-hosted-runner-5mmzk-8rb7f","uid":"301f0ac6-ca28-4aef-82f1-6aeef245cc43","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"450589736"}, "reason": "PodCreated"}
2023-03-31T13:36:50Z	INFO	runnerpod	Unregistration started before runner ID is assigned and the runner stopped before obtaining ID within registration timeout. Perhaps the runner successfully ran the job and stopped normally before the runner ID becomes visible via GitHub API? Perhaps the runner pod was terminated by anyone other than ARC? Was it OOM killed? Marking unregistration as completed anyway because there's nothing ARC can do.	{"runnerpod": "actions-runner-system/org-github-self-hosted-runner-5mmzk-l9zk4", "registrationTimeout": "10m0s"}

Whole Runner Pod Logs

Requesting token at 'https://api.github.com/orgs/unzercorp/actions/runners/registration-token'

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication

Http response code: NotFound from 'POST https://api.github.com/actions/runner-registration'
{"message":"Not Found","documentation_url":"https://docs.github.com/rest"}
Response status code does not indicate success: 404 (Not Found).
.path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Starting Runner listener with startup type: service
Started listener process, pid: 72
Started running service
An error occurred: Not configured. Run config.(sh/cmd) to configure the runner.
Runner listener exited with error code 1
Runner listener exit with terminated error, stop the service, no retry needed.

Additional Context

No response

Start with the RunnerDeployment and HorizontalRunnerAutoscaler YAML, then compare the GitHub App credential setup against docs/authenticating-to-the-github-api.md. Use the controller logs to trace registration-token creation and runner pod replacement; done means the organization runner registers successfully and no longer recreates indefinitely.