Cannot scale from zero with TotalNumberOfQueuedAndInProgressWorkflowRuns metric

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.5

Helm Chart Version

0.23.4

CertManager Version

1.12.1

Deployment Method

Helm

cert-manager installation

Helm install via official charts cert-manager

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
  creationTimestamp: "2023-08-18T07:27:45Z"
  generation: 229
  name: acs-deployment
  namespace: default
  resourceVersion: "5726615"
  uid: 5bb3adab-71f2-4e38-980d-e606438f5822
spec:
  effectiveTime: null
  replicas: 1
  selector: null
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    spec:
      dockerdContainerResources: {}
      dockerdWithinRunnerContainer: true
      image: summerwind/actions-runner-dind:ubuntu-22.04
      repository: Alfresco/acs-deployment
      resources:
        limits:
          cpu: 1750m
          memory: 7Gi
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
  creationTimestamp: "2023-08-24T10:27:08Z"
  generation: 13
  name: acs-deployment-autoscaler
  namespace: default
  resourceVersion: "5727811"
  uid: bd8c52a3-7b47-4eff-b75d-0ea820615d60
spec:
  maxReplicas: 20
  metrics:
  - scaleDownAdjustment: 1
    scaleDownThreshold: "0.3"
    scaleUpAdjustment: 5
    scaleUpThreshold: "0.75"
    type: PercentageRunnersBusy
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  minReplicas: 0
  scaleTargetRef:
    kind: RunnerDeployment
    name: acs-deployment
status:
  desiredReplicas: 11
  lastSuccessfulScaleOutTime: "2023-08-29T13:16:04Z"

To Reproduce

1. Wait for no jobs running so runner deployment get scaled to zero
2. Trigger new workflows

Describe the bug

When zero replicas are currently active, autoscaling is not triggered because TotalNumberOfQueuedAndInProgressWorkflowRuns is always reporting zero pending jobs, despite there are queued workflows waiting to be picked up by a runner.

When this happens, controller clearly show that no workflows are queued for it:

2023-08-29T13:11:32Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "default", "kind": "runnerdeployment", "name": "acs-deployment", "horizontal_runner_autoscaler": "acs-deployment-autoscaler"}

Describe the expected behavior

In a few minutes new replicas should started to begin executing the pending workflows

Whole Controller Logs

https://gist.github.com/gionn/d6abb20e8ce463a2978bc6a549531400

Whole Runner Pod Logs

n/a

Additional Context

No response

Start by reading TROUBLESHOOTING.md and the HorizontalRunnerAutoscaler configuration using TotalNumberOfQueuedAndInProgressWorkflowRuns. Reproduce the zero-replica case from the issue and inspect the linked controller logs. Done means queued workflows cause replicas to start when the RunnerDeployment is scaled to zero, while existing scaling behavior remains intact.