TotalNumberOfQueuedAndInProgressWorkflowRuns does not scale when workflow run is cancelled due to concurrency rules

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.11.0

Deployment Method

Helm

cert-manager installation

Bug unrelated to cert-manager, which looks fine.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  annotations:
    meta.helm.sh/release-name: runner-large
    meta.helm.sh/release-namespace: arc
  creationTimestamp: "2023-05-10T11:25:39Z"
  generation: 656
  labels:
    app.kubernetes.io/managed-by: Helm
  name: runner-large
  namespace: arc
  resourceVersion: "41545469"
  uid: 238b94dd-5c1d-4fb5-a975-43bd23caf204
spec:
  dockerdWithinRunnerContainer: true
  image: xxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/z8o9m4l5/system-components/mirrors/summerwind/actions-runner-dind:v2.304.0-ubuntu-20.04
  labels:
  - k8s-large
  organization: openplayuk
  replicas: 0
  selector:
    matchLabels:
      app: runner-large
  serviceName: runner-large
  template:
    metadata:
      annotations:
        ad.datadoghq.com/runner.logs: |
          [{
            "service": "actions-runner",
            "source": "runner"
          }]
      labels:
        app: runner-large
        tags.datadoghq.com/env: runners
        tags.datadoghq.com/service: actions-runner
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - -c
        - docker login -u AWS -p $(cat /ecr/pass) xxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com
          && entrypoint-dind.sh
        command:
        - sh
        env:
        - name: DISABLE_RUNNER_UPDATE
          value: "true"
        - name: WAIT_FOR_DOCKER
          value: "false"
        name: runner
        resources:
          limits:
            cpu: 4000m
            memory: 16Gi
          requests:
            cpu: 3000m
            memory: 12Gi
        volumeMounts:
        - mountPath: /ecr
          name: ecr
      initContainers:
      - args:
        - -c
        - aws ecr get-login-password --region eu-west-1 > /ecr/pass
        command:
        - sh
        image: public.ecr.aws/aws-cli/aws-cli:2.11.4
        name: get-ecr-pass
        volumeMounts:
        - mountPath: /ecr
          name: ecr
      nodeSelector:
        karpenter.sh/provisioner-name: arc-large
      securityContext:
        fsGroup: 1000
      serviceAccountName: arc-service-account
      volumes:
      - emptyDir:
          medium: Memory
        name: ecr
status:
  availableReplicas: 0
  desiredReplicas: 0
  readyReplicas: 0
  replicas: 0
  updatedReplicas: 0

---------------------------------------------------------

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: runner-large
    meta.helm.sh/release-namespace: arc
  creationTimestamp: "2023-05-10T11:25:39Z"
  generation: 3
  labels:
    app.kubernetes.io/managed-by: Helm
  name: runner-large-autoscaler
  namespace: arc
  resourceVersion: "41545255"
  uid: ba143253-8ca4-465d-8e90-ea2282f6f49a
spec:
  maxReplicas: 800
  metrics:
    - repositoryNames:
        - blueprint-unified
        - core-docker-image
        - kubescanner
      type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 600
  scaleTargetRef:
    kind: RunnerSet
    name: runner-large
status:
  desiredReplicas: 0
  lastSuccessfulScaleOutTime: "2023-05-24T14:22:50Z"

To Reproduce

1. Add a workflow with this configuration

---------------------------------------------
on:
  pull_request:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
  group: '${{ github.workflow }}.${{ github.head_ref }}'
  cancel-in-progress: true
--------------------------------------------

2. Open 2 PRs against from 2 different source branches, eg `feature1` and `feature2` against same branch e.g `develop`.

3. Push on PR1 - workflow run 1 triggered
4. Push on PR2 - workflow run 2 triggered
5. Push on PR1 - workflow run 1 is cancelled due to concurrency rules - workflow 3 is queued
6. Push on PR2 - workflow run 2 is cancelled due to concurrency rules, workflow run 4 is queued

Describe the bug

The autoscaler won't calculate pods for runs 3 and 4 and they will get stuck.
As soon as all the pods that were launched for runs 1 and 2 are completed, the autoscaler will calculate 0 desired pods, although there are queued jobs for runs 3 and 4.

Logs:
Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns

{
--
horizontal_runner_autoscaler | runner-large-autoscaler
kind | runnerset
logger | horizontalrunnerautoscaler
name | runner-large
namespace | arc
severity | debug
ts | 2023-05-25T07:21:57Z
workflow_runs_completed | 0
workflow_runs_in_progress | 0
workflow_runs_queued | 0
workflow_runs_unknown | 0
}

The jobs do have a k8s-large label and they are picked up fine on other scenarios.

Describe the expected behavior

The autoscaler should identify that there are queued runs and jobs and it should scale up.
I checked with github api and verified that runs 3 and 4 are returned as queued, same do their jobs.
I redeployed the controller and it didn't scale again, thus I don't think it's a cache thing.
Sync period is 3s, but I also tried with 1min, it didn't help.
I suspect that the issue is somewhere here but couldn't find out where exactly.

Whole Controller Logs

Nothing special, just

Calculated desired replicas of 0
Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns

Whole Runner Pod Logs

https://gist.github.com/tania-pets/2c17a3ba7170c49df0e419de1eed0ecd

Additional Context

Apart from the runner-large RunnerSet that I post above, I have 2 more runnersets runner-small, runner-xlarge similar to the runner-large and 2 more HorizontalAutoscalers accordingly runner-small-autoscaler, runner-xlarge-autoscaler that target to each runnerset (one to one)

Start with controllers/actions.summerwind.net/autoscaling.go around line 91 and reproduce the concurrency scenario described in the issue. Compare the GitHub API's queued runs and jobs with the controller's logged counts after earlier runs are cancelled. Done means TotalNumberOfQueuedAndInProgressWorkflowRuns detects the queued runs and scales the RunnerSet above zero.