Runners will never be created for cron job if one Github event is missed

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.12.2

Deployment Method

Helm

cert-manager installation

Install cert manager from the following helm file:

releases:
- name: cert-manager
  namespace: cert-manager
  createNamespace: true
  chart: jetstack/cert-manager
  version: 1.12.2
  wait: true
  set:
  - name: installCRDs
    value: true
  - name: prometheus.enabled
    value: false
  # The cert-manager-webhook deployment uses port 10250 by default,
  # which is also used for kubelet on the Fargate pods.
  # See https://github.com/cert-manager/cert-manager/issues/3237
  - name: webhook.securePort
    value: 10260

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: panda-workflow-sample-autoscaler
spec:
  maxReplicas: 10
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: panda-workflow-sample-runner
  scaleUpTriggers:
  - duration: 30m
    githubEvent:
      workflowJob: {}
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: panda-workflow-sample-runner
spec:
  replicas: 1
  template:
    metadata:
      labels:
        fargate: "true"
    spec:
      dockerEnabled: false
      dockerdContainerResources:
        limits:
          cpu: "0.5"
          memory: 1Gi
      image: ****
      labels:
      - bjorn-panda-workflow-sample
      - aws
      - ap-northeast-1
      - ubuntu-latest
      - eks
      repository: ****/panda-workflow-sample
      resources:
        limits:
          cpu: "2"
          memory: 3.5Gi
        requests:
          cpu: "2"
          memory: 3.5Gi

To Reproduce

1. Create a crontab job running with following feature:
   1.1 Runs every 10 minutes 
   1.2 Using concurrency to ensure that only a single job to run at a time
2. Change Webhook Payload URL to a wrong address on purpose.
3. After the first job queued, change Webhook Payload URL to the correct one.
   Using this operation to simulate error during sending "workflow_job.queued" event.
4. Waiting the second job to appear in actions console
5. Cancel the first job

Describe the bug

Three events were sent during above operation:

  cd390600-2fb0-11ee-88ee-d7a698917f31  workflow_job.completed … 2023-07-31 22:44:55
  cd9a38d0-2fb0-11ee-9689-6b1576fe5274  workflow_job.queued … 2023-07-31 22:44:54
  6900eca0-2fad-11ee-8372-d5e1b093c17f  workflow_job.queued… 2023-07-31 22:20:35 (failed to sent this event)

The above timezone is GMT+8.
Though the second job would be in "Queue" status, there is no Github Actions runner to pickup this job.
No runner pods is created on EKS Fargate, and the DESIRED is 0 in RunnerDeployment.

Describe the expected behavior

There should be one runner to pick up the second job.

Whole Controller Logs

2023-07-31T14:40:56Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:40:56Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:41:58Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:41:58Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:42:59Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:42:59Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:44:00Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:44:00Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:45:02Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:45:02Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:46:03Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:46:03Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-workflow-sample-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}
2023-07-31T14:47:05Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "actions-runner/panda-cloud-infra-autoscaler", "suggested": 0, "reserved": 0, "min": 0, "max": 10}

Whole Runner Pod Logs

Since no runner pod was created, no logs here.

Additional Context

When the first job is cancelled, Github sends a "workflow_job.completed" event of the first job to Webhook service, it also sends a "workflow_job.queued" event of the second job to Webhook service.
These 2 events are sent almost at the same time, and the order is not guaranteed.

I think the event workflow_job.completed would cause the scaling counter -1 and the event workflow_job.queued would cause the scaling counter +1.
Is there no change after the effects of the two are superimposed?

For crontab job, no subsequent job will be run without manual operation since there is no runner forever.

Start by tracing the HorizontalRunnerAutoscaler handling for workflow_job webhook events and the desired-replica calculation shown in the controller logs. Reproduce the missed queued event and near-simultaneous completed and queued events from the report; done means a queued second job causes a runner to be created instead of leaving the RunnerDeployment at zero.