Unable to get registration_token 404 Error (GET request instead of POST)

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

latest

Helm Chart Version

v3.11.3

CertManager Version

latest

Deployment Method

Helm

cert-manager installation

Steps in - https://github.com/actions/actions-runner-controller/blob/master/docs/quickstart.md

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  replicas: 1
  template:
    spec:
      repository: myorg/myrepo

To Reproduce

1. Install cert manager
2. Add PAT token
3. Helm deployment

helm upgrade --install --namespace actions-runner-system --create-namespace\
  --set=authSecret.create=true\
  --set=authSecret.github_token="ghp_xxxxxPAT"\
  --set=githubURL="https://githubesurl/api/v3"\
  --set=githubUploadURL="https://githubesurl/api/uploads"\
  --set=runnerGithubURL="https://githubesurl"\
  --set=additionalVolumeMounts[0].mountPath="/usr/local/share/ca-certificates"\
  --set=additionalVolumeMounts[0].name=ghe-cert\
  --set additionalVolumes[0].name=ghe-cert\
  --set additionalVolumes[0].secret.secretName=ghecert\
  --wait actions-runner-controller actions-runner-controller/actions-runner-controller

4. Create a runner against the repo

kubectl apply -f runnerdeployment.yaml  (example-runnerdeploy)

Expected runners in RUNNING status

Describe the bug

I am setting up arcs on Azure AKS for self-hosted runners to be used from github enterprise server (on-prem) .

I have performed the steps as in https://github.com/actions/actions-runner-controller/blob/master/docs/quickstart.md

But the pods do not come up. The controller log shows the following error- failed to create registration token: GET https://gitesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found

 2023-05-01T18:45:31Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-rq6d4-gq9c2","uid":"cd5dd01e-8213-44a8-abb3-99b2f702c2fa","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"697203"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-01T18:45:31Z    ERROR   runner  Failed to get new registration token    {"runner": "github-runner-api-123-bzc6p-2mwgq", **_"error": "failed to create registration token: GET https://gitesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found []"_**}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).updateRegistrationToken
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:456
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerCreation
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:292
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:147
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235

I am expecting a POST request here not GET. Is the issue because of that?

Following steps I have had to do additionally to make it work with github enterprise.

1. Add additional volume and volume mount for certificate (--set=additionalVolumeMounts[0].name=gh-crt ....)
2. Add host alias in deployment.
3. Set env for GITHUB_ENTERPRISE_URL=https://enterpriseurl
4. set env for certificate SSL_CERT_DIR=/usr/share/ca-certificates
5. Add additional params for step 1 (helm upgrade --install --namespace actions-runner-system --create-namespace ...)
   . --set=githubURL="https://githubesurl/api/v3"
   --set=githubUploadURL="https://githubesurl/api/uploads"
   --set=runnerGithubURL="https://githubesurl"\

Does any of it affect the call? I am assuming GET will not be implemented (a 405 error/unimplemented error) .

Any pointers would be very helpful.

Thanks

Describe the expected behavior

Runner in running status.

Whole Controller Logs

sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-05-02T01:48:34Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-rq6d4-gq9c2","uid":"cd5dd01e-8213-44a8-abb3-99b2f702c2fa","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"697203"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-02T01:48:34Z    ERROR   runner  Failed to get new registration token    {"runner": "github-runner-api-123-bzc6p-2mwgq", "error": "failed to create registration token: GET https://githubesurl/api/v3/repos/myorg/myrepo/actions/runners/registration-token: 404 Not Found []"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).updateRegistrationToken
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:456
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerCreation
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:292
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:147
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-05-02T01:48:34Z    DEBUG   events  Updating registration token failed      {"type": "Warning", "object": {"kind":"Runner","namespace":"default","name":"github-runner-api-123-bzc6p-2mwgq","uid":"d926654a-81f5-4cdf-85e9-4416511b1c27","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"728061"}, "reason": "FailedUpdateRegistrationToken"}
2023-05-02T01:48:48Z    INFO    runnerdeployment        Waiting until the newest runnerreplicaset to be 100% available {"runnerdeployment": "default/github-runner-api", "newest_runnerreplicaset": "default/github-runner-api-t8tjv", "newest_runnerreplicaset_replicas_ready": 0, "newest_runnerreplicaset_replicas_desired": 1, "old_runnerreplicasets_count": 2}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/example-runnerdeploy-h7lk7", "owner": "default/example-runnerdeploy-h7lk7-blvjz", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-rq6d4", "owner": "default/github-runner-api-rq6d4-gq9c2", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-123-bzc6p", "owner": "default/github-runner-api-123-bzc6p-2mwgq", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-t8tjv", "owner": "default/github-runner-api-zl728-fxv8m", "pods": null}
2023-05-02T01:48:56Z    DEBUG   runnerreplicaset        Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "default/github-runner-api-zl728", "owner": "default/github-runner-api-zl728-fxv8m", "pods": null}
2023-05-02T01:48:56Z    INFO    runnerdeployment        Waiting until the newest runnerreplicaset to be 100% available {"runnerdeployment": "default/github-runner-api", "newest_runnerreplicaset": "default/github-runner-api-t8tjv", "newest_runnerreplicaset_replicas_ready": 0, "newest_runnerreplicaset_replicas_desired": 1, "old_runnerreplicasets_count": 2}

Whole Runner Pod Logs

No listener pod

Additional Context

1. K8s is Azure AKS
2. Github is enterprise server on-premise
3. Provided SSL and host alias and set GITHUB_ENTERPRISE_URL and SSL_CERT_DIR
4. Curl to POST method works form jump-host.

Start in controllers/actions.summerwind.net/runner_controller.go, especially updateRegistrationToken around line 456, and review the Helm settings for githubURL, githubUploadURL, and runnerGithubURL. Reproduce the registration-token request against GitHub Enterprise and verify the endpoint and method; done means the token is created and the runner reaches running status.