TotalNumberOfQueuedAndInProgressWorkflowRuns does not scale when waiting jobs get approved if there are other waiting jobs.

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.11.0

Deployment Method

Helm

cert-manager installation

Bug unrelated to cert-manager, which looks fine.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  annotations:
    meta.helm.sh/release-name: runner-large
    meta.helm.sh/release-namespace: arc
  creationTimestamp: "2023-05-10T11:25:39Z"
  generation: 656
  labels:
    app.kubernetes.io/managed-by: Helm
  name: runner-large
  namespace: arc
  resourceVersion: "41545469"
  uid: 238b94dd-5c1d-4fb5-a975-43bd23caf204
spec:
  dockerdWithinRunnerContainer: true
  image: xxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/z8o9m4l5/system-components/mirrors/summerwind/actions-runner-dind:v2.304.0-ubuntu-20.04
  labels:
  - k8s-large
  organization: openplayuk
  replicas: 0
  selector:
    matchLabels:
      app: runner-large
  serviceName: runner-large
  template:
    metadata:
      annotations:
        ad.datadoghq.com/runner.logs: |
          [{
            "service": "actions-runner",
            "source": "runner"
          }]
      labels:
        app: runner-large
        tags.datadoghq.com/env: runners
        tags.datadoghq.com/service: actions-runner
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - -c
        - docker login -u AWS -p $(cat /ecr/pass) xxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com
          && entrypoint-dind.sh
        command:
        - sh
        env:
        - name: DISABLE_RUNNER_UPDATE
          value: "true"
        - name: WAIT_FOR_DOCKER
          value: "false"
        name: runner
        resources:
          limits:
            cpu: 4000m
            memory: 16Gi
          requests:
            cpu: 3000m
            memory: 12Gi
        volumeMounts:
        - mountPath: /ecr
          name: ecr
      initContainers:
      - args:
        - -c
        - aws ecr get-login-password --region eu-west-1 > /ecr/pass
        command:
        - sh
        image: public.ecr.aws/aws-cli/aws-cli:2.11.4
        name: get-ecr-pass
        volumeMounts:
        - mountPath: /ecr
          name: ecr
      nodeSelector:
        karpenter.sh/provisioner-name: arc-large
      securityContext:
        fsGroup: 1000
      serviceAccountName: arc-service-account
      volumes:
      - emptyDir:
          medium: Memory
        name: ecr
status:
  availableReplicas: 0
  desiredReplicas: 0
  readyReplicas: 0
  replicas: 0
  updatedReplicas: 0

---------------------------------------------------------

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: runner-large
    meta.helm.sh/release-namespace: arc
  creationTimestamp: "2023-05-10T11:25:39Z"
  generation: 3
  labels:
    app.kubernetes.io/managed-by: Helm
  name: runner-large-autoscaler
  namespace: arc
  resourceVersion: "41545255"
  uid: ba143253-8ca4-465d-8e90-ea2282f6f49a
spec:
  maxReplicas: 800
  metrics:
    - repositoryNames:
        - blueprint-unified
        - core-docker-image
        - kubescanner
      type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 600
  scaleTargetRef:
    kind: RunnerSet
    name: runner-large
status:
  desiredReplicas: 0
  lastSuccessfulScaleOutTime: "2023-05-24T14:22:50Z"

To Reproduce

1. Create a workflow with a matrix job that requires aproval
2. Run it and approve one of the waiting jobs of the matrix

Describe the bug

When there are jobs that pending approval, the workflow run appears as waiting, even if another waiting job gets approved. As I see it, the autoscaler code cares only for queued or in_progress , so it will never scale up upon an approval and the just approved job will wait forever.

Describe the expected behavior

The runner should scale up when a waiting job gets approved.
The autoscaler should fetch the waiting runs, along with in_progress and queued and check if there are any pending jobs for them.

Whole Controller Logs

Nothing special, just

Calculated desired replicas of 0
Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns

Whole Runner Pod Logs

Unrelated.

Additional Context

No response

Start at the TotalNumberOfQueuedAndInProgressWorkflowRuns autoscaler metric and the code that fetches workflow runs; reproduce the matrix job with approval described in the issue. Done means approving one waiting matrix job causes desired replicas to scale above zero and the controller no longer reports zero while that job is pending.