Running ARC with github_app_id, github_app_installation_id and github_app_private_key on repository or organization on Enterprise Server 3.6 fails to start runner pod configured with RunnerDeployment

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.0

Helm Chart Version

0.22.0

CertManager Version

1.9.1

Deployment Method

Helm

cert-manager installation

I did not install cert-manager myself.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

values.yaml for ARE setup:

githubURL: https://github.my_enterprise_name.com/api/v3
githubUploadURL: https://github.my_enterprise_name.com/api/uploads
runnerGithubURL: https://github.my_enterprise_name.com
scope:
  watchNamespace: my-actions-runner-ns
  singleNamespace: true
authSecret:
  create: true
  github_app_id: my_app_id
  github_app_installation_id: my_installation_id
  github_app_private_key: |
my_private_key

RunnerDeployment yaml:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: my-deployment-name
  namespace: my-actions-runner-ns
spec:
  replicas: 1
  template:
    spec:
      repository: my-org/my-repo
      image: summerwind/actions-runner:ubuntu-20.04

To Reproduce

1. We use Github Enterprise Server 3.6
2. install ARC using values file above. ie
helm upgrade --install --namespace actions-runner-system-dev --create-namespace -f values-dev.yaml --wait my-actions-runner-controller-dev actions-runner-controller/actions-runner-controller
3. setup RunnerDeployment (yaml above):
kubectl create namespace my-actions-runner-ns
kubectl apply -f runnerdeployment-dev.yaml -n my-actions-runner-ns

Describe the bug

Runner pod never starts.
Note that if values.yaml file contains github_token instead of GitHub app, it works: runner pod starts, picks up github actions and processes them.

Describe the expected behavior

Runner pod starts.

Whole Controller Logs

N/A added below.

Whole Runner Pod Logs

no runner logs. Runner does not start.

Additional Context

Instead of above ARC logs I add them here. There is not much (some values are edited as I am not sure about the disclosure):
Defaulted container "manager" out of: manager, kube-rbac-proxy
I0330 18:04:43.809193 1 request.go:682] Waited for 1.001305342s due to client-side throttling, not priority and fairness, request: GET:https://10.212.95.1:443/apis/storage.k8s.io/v1?timeout=32s
2023-03-30T18:04:44Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2023-03-30T18:04:44Z INFO Initializing actions-runner-controller {"version": "v0.27.0", "default-scale-down-delay": "10m0s", "sync-period": "1m0s", "default-runner-image": "summerwind/actions-runner:latest", "default-docker-image": "docker:dind", "common-runnner-labels": null, "leader-election-enabled": true, "leader-election-id": "actions-runner-controller", "watch-namespace": "actions-runner-dev-ns"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-03-30T18:04:44Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-03-30T18:04:44Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-runner-set-pod"}
2023-03-30T18:04:44Z INFO starting manager
2023-03-30T18:04:44Z INFO controller-runtime.webhook.webhooks Starting webhook server
2023-03-30T18:04:44Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-03-30T18:04:44Z INFO controller-runtime.certwatcher Updated current TLS certificate
2023-03-30T18:04:44Z INFO controller-runtime.certwatcher Starting certificate watcher
2023-03-30T18:04:44Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
I0330 18:04:44.820002 1 leaderelection.go:248] attempting to acquire leader lease actions-runner-system-dev/actions-runner-controller...
I0330 18:04:44.844082 1 leaderelection.go:258] successfully acquired lease actions-runner-system-dev/actions-runner-controller
2023-03-30T18:04:44Z DEBUG events kws-actions-runner-controller-dev-6c5576889f-wwqw5_c0debb37-95d8-4f51-8a70-b30cf18a81a2 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"actions-runner-system-dev","name":"actions-runner-controller","uid":"8e115fdd-0c4e-4bab-ab5b-bac45106cfad","apiVersion":"coordination.k8s.io/v1","resourceVersion":"328458245"}, "reason": "LeaderElection"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1alpha1.Runner"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1alpha1.RunnerSet"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "source": "kind source: *v1.PersistentVolume"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.Runner"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1.StatefulSet"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerDeployment"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "source": "kind source: *v1alpha1.HorizontalRunnerAutoscaler"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler"}
2023-03-30T18:04:44Z INFO Starting EventSource {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1.Pod"}
2023-03-30T18:04:44Z INFO Starting Controller {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner"}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-03-30T18:04:44Z INFO Starting workers {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "worker count": 1}
2023-03-30T18:05:08Z INFO runnerdeployment Created runnerreplicaset {"runnerdeployment": "actions-runner-dev-ns/xyz-test-dev", "runnerreplicaset": "xyz-test-dev-lwtc8"}
2023-03-30T18:05:08Z DEBUG runnerreplicaset Created replica(s) {"runnerreplicaset": "actions-runner-dev-ns/xyz-test-dev-lwtc8", "lastSyncTime": null, "effectiveTime": "", "templateHashDesired": "797fc489df", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-03-30T18:05:08Z DEBUG runnerreplicaset Skipped reconcilation because owner is not synced yet {"runnerreplicaset": "actions-runner-dev-ns/xyz-test-dev-lwtc8", "owner": "actions-runner-dev-ns/xyz-test-dev-lwtc8-wzqb8", "pods": null}
2023-03-30T18:05:08Z DEBUG runnerreplicaset Skipped reconcilation because owner is not synced yet {"runnerreplicaset": "actions-runner-dev-ns/xyz-test-dev-lwtc8", "owner": "actions-runner-dev-ns/xyz-test-dev-lwtc8-wzqb8", "pods": null}
2023-03-30T18:05:08Z ERROR runner Failed to get new registration token {"runner": "xyz-test-dev-lwtc8-wzqb8", "error": "failed to create registration token: Post "https://github.enterprise-name.com/api/v3/repos/my-org/my-repo/actions/runners/registration-token\": could not refresh installation id XYZ's token: received non 2xx response status "404 Not Found" when fetching https://api.github.com/app/installations/XYZ/access_tokens"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).updateRegistrationToken
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:446
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerCreation
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:282
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:137
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234
2023-03-30T18:05:08Z DEBUG events Updating registration token failed {"type": "Warning", "object": {"kind":"Runner","namespace":"actions-runner-dev-ns","name":"xyz-test-dev-lwtc8-wzqb8","uid":"5980a0de-1b1d-4d0b-9493-daefdf00b181","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"328458443"}, "reason": "FailedUpdateRegistrationToken"}

Begin with the values.yaml GitHub App authSecret settings and the RunnerDeployment YAML, then compare the controller behavior in the supplied logs with the working github_token path. Done means the Enterprise Server 3.6 RunnerDeployment creates and starts a runner pod that processes GitHub Actions jobs.