Enabling both webhook and PercentageRunnersBusy for HRA leads to excessive runner pod creation

Checks

• I've already read https://github.com/actions-runner-controller/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.26.0

Helm Chart Version

0.21.1

CertManager Version

1.9.1

Deployment Method

Helm

cert-manager installation

cert-manager 1.9.1 was installed using helm.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: test-runner-deployment
spec:
  template:
    spec:
      organization: zephyrproject-rtos
      labels:
      - test-runner
      containers:
      - name: runner
        image: 724087766192.dkr.ecr.us-east-2.amazonaws.com/ci-zephyr-runner-arc-runner:latest
        resources:
          limits:
            cpu: "16.0"
            memory: "32Gi"
          requests:
            cpu: "7.5"
            memory: "12Gi"
        volumeMounts:
        - mountPath: "/repo-cache"
          name: runner-repo-cache
          readOnly: true
      - name: docker
        image: 724087766192.dkr.ecr.us-east-2.amazonaws.com/ci-zephyr-runner-arc-dind:latest
        resources:
          limits:
            cpu: "16.0"
            memory: "32Gi"
          requests:
            cpu: "7.5"
            memory: "12Gi"
        volumeMounts:
        - mountPath: "/repo-cache"
          name: runner-repo-cache
          readOnly: true
      volumes:
      - name: runner-repo-cache
        persistentVolumeClaim:
          claimName: runner-repo-cache-pvc
      nodeSelector:
        InstanceType: spot
      tolerations:
      - key: "spotInstance"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: test-runner-deployment-autoscaler
spec:
  scaleTargetRef:
    kind: RunnerDeployment
    name: test-runner-deployment
  minReplicas: 0
  maxReplicas: 100
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    duration: "15m"
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.3'
    scaleUpAdjustment: 1
    scaleDownAdjustment: 1

To Reproduce

1. Apply the above resource definitions (create an HRA with both `workflowJob` scale-up trigger and `PercentageRunnersBusy` metric).
2. Queue 15 jobs (a workflow that launches 15 jobs)
3. Observe the controller create runner pods up to the `maxReplicas` count (in case of the provided HRA, 100).

Describe the bug

Enabling both webhook (workflowJob scale-up trigger) and PercentageRunnersBusy metric for HRA leads to excessive runner pod creation (in my case, 100 pods = maxReplicas) when many jobs (in my case, 15) are simultaneously queued:

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: test-runner-deployment-autoscaler
spec:
  scaleTargetRef:
    kind: RunnerDeployment
    name: test-runner-deployment
  minReplicas: 0
  maxReplicas: 100
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    duration: "15m"
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.3'
    scaleUpAdjustment: 1
    scaleDownAdjustment: 1

Describe the expected behavior

When 15 jobs are queued, the runner pods are only scaled up to 15 + alpha (from PercentageRunnersBusy), not 100 (maxReplicas).

Whole Controller Logs

Some interesting excerpts:

2022-10-28T11:16:49Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "default/test-runner-deployment-4zwql", "owner": "default/test-runner-deployment-4zwql-wf6j9", "pods": null}
2022-10-28T11:16:49Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Suggested desired replicas of 15 by PercentageRunnersBusy	{"replicas_desired_before": 16, "replicas_desired": 15, "num_runners": 12, "num_runners_registered": 0, "num_runners_busy": 0, "num_terminating_busy": 1, "namespace": "default", "kind": "runnerdeployment", "name": "test-runner-deployment", "horizontal_runner_autoscaler": "test-runner-deployment-autoscaler", "enterprise": "", "organization": "zephyrproject-rtos", "repository": ""}
2022-10-28T11:16:49Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 17	{"horizontalrunnerautoscaler": "default/test-runner-deployment-autoscaler", "suggested": 15, "reserved": 2, "min": 0, "max": 100}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "c77c9f08-a13f-453a-9516-a9ba263b6284", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "c77c9f08-a13f-453a-9516-a9ba263b6284", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "default/test-runner-deployment-4zwql", "owner": "default/test-runner-deployment-4zwql-wf6j9", "pods": null}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "6a27cde3-22de-4029-8156-115a59ec2ddc", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:49Z	INFO	runnerdeployment-resource	validate resource to be updated	{"name": "test-runner-deployment"}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "6a27cde3-22de-4029-8156-115a59ec2ddc", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "92906dc0-2188-41de-8888-66b0f3825f36", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "92906dc0-2188-41de-8888-66b0f3825f36", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "51a2871a-e102-4a81-bd0f-54bc3297b2fa", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "code": 200, "reason": "", "UID": "51a2871a-e102-4a81-bd0f-54bc3297b2fa", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "eb8ec9de-2960-4e35-929a-6e7a11348011", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:49Z	INFO	runner-resource	validate resource to be updated	{"name": "test-runner-deployment-4zwql-hjs9d"}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "eb8ec9de-2960-4e35-929a-6e7a11348011", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "80ad54cd-e4b0-4746-9e69-3b1f1d52a94f", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2022-10-28T11:16:49Z	INFO	runnerreplicaset-resource	validate resource to be updated	{"name": "test-runner-deployment-4zwql"}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "code": 200, "reason": "", "UID": "80ad54cd-e4b0-4746-9e69-3b1f1d52a94f", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "e4533542-71c0-4019-94f3-abf9fe0ddafa", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:49Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "e4533542-71c0-4019-94f3-abf9fe0ddafa", "allowed": true}
2022-10-28T11:16:49Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "default/test-runner-deployment-4zwql", "owner": "default/test-runner-deployment-4zwql-wf6j9", "pods": null}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "a4ffabe9-c625-483d-933d-f5c9e58fded3", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:50Z	INFO	runnerdeployment-resource	validate resource to be updated	{"name": "test-runner-deployment"}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "a4ffabe9-c625-483d-933d-f5c9e58fded3", "allowed": true}
2022-10-28T11:16:50Z	DEBUG	actions-runner-controller.runnerdeployment	Updated runnerreplicaset due to spec change	{"runnerdeployment": "default/test-runner-deployment", "currentDesiredReplicas": 15, "newDesiredReplicas": 16, "currentEffectiveTime": "2022-10-28 11:16:49 +0000 UTC", "newEffectiveTime": "2022-10-28 11:16:49 +0000 UTC"}
2022-10-28T11:16:50Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "default/test-runner-deployment-4zwql", "owner": "default/test-runner-deployment-4zwql-cn6fv", "pods": null}
2022-10-28T11:16:50Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "default/test-runner-deployment-4zwql", "owner": "default/test-runner-deployment-4zwql-7zkzh", "pods": null}
2022-10-28T11:16:50Z	INFO	actions-runner-controller.runner	Updated registration token	{"runner": "test-runner-deployment-4zwql-qvs6d", "repository": ""}
2022-10-28T11:16:50Z	DEBUG	events	Normal	{"object": {"kind":"Runner","namespace":"default","name":"test-runner-deployment-4zwql-qvs6d","uid":"1e32c7c9-2d20-4671-9483-4debccebd57c","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"1042339"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "0e2da8cf-c718-4aa9-ab65-1eb2d720e034", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "0e2da8cf-c718-4aa9-ab65-1eb2d720e034", "allowed": true}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "da688fcd-2a0b-4a84-bec8-ac978db0e165", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:50Z	INFO	runner-resource	validate resource to be updated	{"name": "test-runner-deployment-4zwql-cn6fv"}
2022-10-28T11:16:50Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "da688fcd-2a0b-4a84-bec8-ac978db0e165", "allowed": true}
2022-10-28T11:16:50Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Suggested desired replicas of 16 by PercentageRunnersBusy	{"replicas_desired_before": 17, "replicas_desired": 16, "num_runners": 12, "num_runners_registered": 0, "num_runners_busy": 0, "num_terminating_busy": 1, "namespace": "default", "kind": "runnerdeployment", "name": "test-runner-deployment", "horizontal_runner_autoscaler": "test-runner-deployment-autoscaler", "enterprise": "", "organization": "zephyrproject-rtos", "repository": ""}
2022-10-28T11:16:50Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 18	{"horizontalrunnerautoscaler": "default/test-runner-deployment-autoscaler", "suggested": 16, "reserved": 2, "min": 0, "max": 100}
[...]
2022-10-28T11:16:53Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Suggested desired replicas of 87 by PercentageRunnersBusy	{"replicas_desired_before": 88, "replicas_desired": 87, "num_runners": 30, "num_runners_registered": 0, "num_runners_busy": 0, "num_terminating_busy": 1, "namespace": "default", "kind": "runnerdeployment", "name": "test-runner-deployment", "horizontal_runner_autoscaler": "test-runner-deployment-autoscaler", "enterprise": "", "organization": "zephyrproject-rtos", "repository": ""}
2022-10-28T11:16:53Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 94	{"horizontalrunnerautoscaler": "default/test-runner-deployment-autoscaler", "suggested": 87, "reserved": 7, "min": 0, "max": 100}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "7ddaa57d-73ef-4324-804a-9e39d3c4d2b7", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "7ddaa57d-73ef-4324-804a-9e39d3c4d2b7", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "6bf91cda-f7bd-43f7-bba5-ce9a6cb01feb", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-28T11:16:53Z	INFO	runner-resource	validate resource to be created	{"name": "test-runner-deployment-4zwql-7fp4j"}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "6bf91cda-f7bd-43f7-bba5-ce9a6cb01feb", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "912f6584-44ef-4f8d-90ac-e82071e0db60", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "code": 200, "reason": "", "UID": "912f6584-44ef-4f8d-90ac-e82071e0db60", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "UID": "b0bbc118-ab55-48f5-8a3d-a048d8699f3d", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerreplicasets"}}
2022-10-28T11:16:53Z	INFO	runnerreplicaset-resource	validate resource to be updated	{"name": "test-runner-deployment-4zwql"}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset", "code": 200, "reason": "", "UID": "b0bbc118-ab55-48f5-8a3d-a048d8699f3d", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	actions-runner-controller.runnerdeployment	Updated runnerreplicaset due to spec change	{"runnerdeployment": "default/test-runner-deployment", "currentDesiredReplicas": 82, "newDesiredReplicas": 88, "currentEffectiveTime": "2022-10-28 11:16:52 +0000 UTC", "newEffectiveTime": "2022-10-28 11:16:52 +0000 UTC"}
2022-10-28T11:16:53Z	INFO	actions-runner-controller.runner	Updated registration token	{"runner": "test-runner-deployment-4zwql-224sq", "repository": ""}
2022-10-28T11:16:53Z	DEBUG	events	Normal	{"object": {"kind":"Runner","namespace":"default","name":"test-runner-deployment-4zwql-224sq","uid":"64811958-5728-44ae-9afd-20695e4f7a35","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"1042755"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "d9cf541c-40e1-42dc-82d2-a20f0ae3a544", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "d9cf541c-40e1-42dc-82d2-a20f0ae3a544", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "UID": "7d6e0ef4-378e-4f37-b4bd-1e77ef138b27", "kind": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runnerdeployments"}}
2022-10-28T11:16:53Z	INFO	runnerdeployment-resource	validate resource to be updated	{"name": "test-runner-deployment"}
2022-10-28T11:16:53Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment", "code": 200, "reason": "", "UID": "7d6e0ef4-378e-4f37-b4bd-1e77ef138b27", "allowed": true}
2022-10-28T11:16:53Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Suggested desired replicas of 93 by PercentageRunnersBusy	{"replicas_desired_before": 94, "replicas_desired": 93, "num_runners": 31, "num_runners_registered": 0, "num_runners_busy": 0, "num_terminating_busy": 1, "namespace": "default", "kind": "runnerdeployment", "name": "test-runner-deployment", "horizontal_runner_autoscaler": "test-runner-deployment-autoscaler", "enterprise": "", "organization": "zephyrproject-rtos", "repository": ""}
2022-10-28T11:16:53Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 100	{"horizontalrunnerautoscaler": "default/test-runner-deployment-autoscaler", "suggested": 93, "reserved": 7, "min": 0, "max": 100}

For full log, please see https://gist.github.com/stephanosio/68e9604948d827cf5a86ba8b2b152665

The errors at the end of log ("OOM killed?", ...) are because I deleted the deployment to quickly get rid of the 100 runner pods, so they can be ignored.

Whole Runner Pod Logs

100 pods are created, so not really feasible to provide the logs for every one of them ...

Additional Context

I have also tried:

• with the upstream runner images (summerwind/actions-runner:latest for runner and docker:dind for docker) and the same behaviour (runner pods maxing out) was observed.
• without the runner-repo-cache volume, it did not make any difference.

p.s. the reason that I am trying to enable both webhook and PercentageRunnersBusy, as opposed to only using webhook, is that runners sometimes get terminated after start-up for some unknown reason and some jobs end up remaining in the queued status indefinitely, and using PercentageRunnersBusy will at least ensure that such jobs are scheduled by launching additional runners if all runners are busy. Is there a better existing way to address this issue -- for instance, something that periodically checks if there are queued jobs and launches additional runners? -- UPDATE: For now, I am just using TotalNumberOfQueuedAndInProgressWorkflowRuns-only configuration. That seems to be the most stable and robust solution (PercentageRunnersBusy was over-allocating the runners too much).

p.s. 2. webhook + TotalNumberOfQueuedAndInProgressWorkflowRuns configuration seems to have the same issue.

Start by tracing HorizontalRunnerAutoscaler reconciliation for the workflowJob scale-up trigger and PercentageRunnersBusy metric, using the supplied controller logs to follow desired and reserved replica calculations. Reproduce the 15-job configuration and verify that completion limits scaling near the queued job count plus the metric adjustment rather than maxReplicas.