(Kustomize) Webhook Server Not Scaling

Controller Version

v0.25.2

Helm Chart Version

No response

CertManager Version

v1.9.1

Deployment Method

Kustomize

cert-manager installation

• Installed via their official documentation. Runners work fine when using polling HRA.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/actions-runner-controller/actions-runner-controller/config//default?ref=v0.25.2
  - github.com/actions-runner-controller/actions-runner-controller/config//github-webhook-server?ref=v0.25.2
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: default-runner
spec:
  template:
    spec:
      nodeSelector:
        node: "runner"
      tolerations:
      - effect: NoSchedule
        key: node
        value: runner
        operator: Equal
      organization: moes
      serviceAccountName: runner
      imagePullPolicy: Always
      labels:
        - default

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: default-runner
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  minReplicas: 1
  maxReplicas: 20
  scaleTargetRef:
    name: default-runner
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    amount: 1
    duration: "10m"
---
name: Test
on:
  push:
    branches: [test]

jobs:
  first-job:
    runs-on: default
    steps:
      ### Checkout Code
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: sleep 180

  second-job:
    runs-on: default
    steps:
      ### Checkout Code
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: sleep 180

To Reproduce

1. create ingress for webhook-server
2. verify working by seeing 'webhook server is running'
3. add webhook to github repo
4. run a github action with two jobs
5. only one of them runs on the already running runner

Describe the bug

Webhook server is not scaling runners. Polling works fine but when switching over to use the webhook server, nothing happens.

• Webhook passed in github (green check)
• Sending: push, pull_request, check run, workflow jobs
• No logs on the webhook server
• No errors on the controller

Describe the expected behavior

Webhook server scales up and both jobs run at the same time.

Controller Logs

2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "e8058ffc-f65a-444b-a37b-3f1b4ebfb60a", "allowed": true}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "94bd66c4-1c36-445d-9ce3-5c7fa9ba170b", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-08-19T17:37:29Z	INFO	runner-resource	validate resource to be created	{"name": "default-runner-kbzhf-cks54"}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "94bd66c4-1c36-445d-9ce3-5c7fa9ba170b", "allowed": true}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "08e65e99-5b06-4bef-9564-cfcfc8d406bf", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-08-19T17:37:29Z	INFO	runner-resource	validate resource to be updated	{"name": "default-runner-kbzhf-cnc5r"}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "08e65e99-5b06-4bef-9564-cfcfc8d406bf", "allowed": true}
2022-08-19T17:37:29Z	INFO	actions-runner-controller.runner	Removed finalizer	{"runner": "actions-runner-system/default-runner-kbzhf-cnc5r"}
2022-08-19T17:37:29Z	DEBUG	actions-runner-controller.runnerreplicaset	Created replica(s)	{"runnerreplicaset": "actions-runner-system/default-runner-kbzhf", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "764bcc86b7", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2022-08-19T17:37:29Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/default-runner-kbzhf", "owner": "actions-runner-system/default-runner-kbzhf-cks54", "pods": null}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "cac0ac6a-a124-4d85-b40f-105c41f5b861", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "cac0ac6a-a124-4d85-b40f-105c41f5b861", "allowed": true}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "7cde435c-f6bc-4451-be15-454be15d2361", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-08-19T17:37:29Z	INFO	runner-resource	validate resource to be updated	{"name": "default-runner-kbzhf-cks54"}
2022-08-19T17:37:29Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "7cde435c-f6bc-4451-be15-454be15d2361", "allowed": true}
2022-08-19T17:37:29Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/default-runner-kbzhf", "owner": "actions-runner-system/default-runner-kbzhf-cks54", "pods": null}
2022-08-19T17:37:29Z	INFO	actions-runner-controller.runnerpod	Runner pod has been stopped with a successful status.	{"runnerpod": "actions-runner-system/default-runner-kbzhf-cnc5r"}
2022-08-19T17:37:29Z	INFO	actions-runner-controller.runnerpod	Runner pod is marked as already unregistered.	{"runnerpod": "actions-runner-system/default-runner-kbzhf-cnc5r"}
2022-08-19T17:37:30Z	INFO	actions-runner-controller.runner	Updated registration token	{"runner": "default-runner-kbzhf-cks54", "repository": ""}
2022-08-19T17:37:30Z	DEBUG	events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"default-runner-kbzhf-cks54","uid":"5c946ded-5d33-4688-8c2a-2113d288fc40","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"48069792"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2022-08-19T17:37:30Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/default-runner-kbzhf", "owner": "actions-runner-system/default-runner-kbzhf-cks54", "pods": null}
2022-08-19T17:37:32Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-system/default-runner-kbzhf", "owner": "actions-runner-system/default-runner-kbzhf-cks54", "pods": null}
2022-08-19T17:37:35Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-runner-set-pod", "UID": "6a5a2156-b60d-4ef6-aada-d430e8ea8fa0", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2022-08-19T17:37:35Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-runner-set-pod", "code": 200, "reason": "", "UID": "6a5a2156-b60d-4ef6-aada-d430e8ea8fa0", "allowed": true}
2022-08-19T17:37:40Z	INFO	actions-runner-controller.runner	Created runner pod	{"runner": "actions-runner-system/default-runner-kbzhf-cks54", "repository": ""}
2022-08-19T17:37:40Z	DEBUG	events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"default-runner-kbzhf-cks54","uid":"5c946ded-5d33-4688-8c2a-2113d288fc40","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"48069803"}, "reason": "PodCreated", "message": "Created pod 'default-runner-kbzhf-cks54'"}
2022-08-19T17:37:42Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "actions-runner-system/default-runner", "suggested": 1, "reserved": 0, "min": 1, "max": 20}
2022-08-19T17:37:45Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-runner-set-pod", "UID": "983775c5-d4f6-4fac-9d7b-61f9a7f98dc9", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2022-08-19T17:37:45Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-runner-set-pod", "code": 200, "reason": "", "UID": "983775c5-d4f6-4fac-9d7b-61f9a7f98dc9", "allowed": true}
2022-08-19T17:37:50Z	INFO	actions-runner-controller.runner	Failed to create pod due to AlreadyExists error. Probably this pod has been already created in previous reconcilation but is still not in the informer cache. Will retry on pod created. If it doesn't repeat, there's no problem	{"runner": "actions-runner-system/default-runner-kbzhf-cks54"}
2022-08-19T17:37:50Z	DEBUG	actions-runner-controller.runner	Runner appears to have been registered and running.	{"runner": "actions-runner-system/default-runner-kbzhf-cks54", "podCreationTimestamp": "2022-08-19 17:37:40 +0000 UTC"}
2022-08-19T17:38:44Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "actions-runner-system/default-runner", "suggested": 1, "reserved": 0, "min": 1, "max": 20}
Logs from 8/19/2022, 11:34:18 AM

Runner Pod Logs

Logs from the webhook server:

I0811 18:42:49.683121       1 main.go:177] Valid token audiences: 
I0811 18:42:49.683180       1 main.go:259] Generating self signed cert as no cert is provided
I0811 18:42:49.889764       1 main.go:309] Starting TCP socket on 0.0.0.0:8443
I0811 18:42:49.890266       1 main.go:316] Listening securely on 0.0.0.0:8443

Additional Context

Polling is still enabled with -sync-period=1m, does this need to be disabled when using webhooks?

Start with the Kustomize resources for default and github-webhook-server at v0.25.2, then inspect the webhook server entry point shown in main.go and reproduce the two-job workflow. Compare webhook and polling behavior using the supplied controller and webhook logs; done means a webhook event scales the HRA to two runners and both jobs run concurrently.