ARC Metrics not showing new metrics from 0.26.0

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.0

Helm Chart Version

0.22.0

CertManager Version

No response

Deployment Method

Helm

cert-manager installation

Not using cert-manager

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  creationTimestamp: '2023-02-13T21:55:42Z'
  generation: 4
  managedFields:
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:template:
            f:spec:
              f:automountServiceAccountToken: {}
              f:dockerEnabled: {}
              f:dockerdWithinRunnerContainer: {}
              f:enterprise: {}
              f:env: {}
              f:group: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:imagePullSecrets: {}
              f:labels: {}
              f:nodeSelector:
                f:pool: {}
      manager: Terraform
      operation: Apply
      time: '2023-02-13T21:55:42Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:effectiveTime: {}
          f:replicas: {}
      manager: manager
      operation: Update
      time: '2023-02-13T22:38:55Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:availableReplicas: {}
          f:desiredReplicas: {}
          f:readyReplicas: {}
          f:replicas: {}
          f:updatedReplicas: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-02-13T22:48:54Z'
  name: build-general-linux-dev
  namespace: actions-runner-system-build
  resourceVersion: '13021989'
  uid: 4020a46a-d3fc-4a27-88ae-07108b4a5d0c
  selfLink: >-
    /apis/actions.summerwind.dev/v1alpha1/namespaces/actions-runner-system-build/runnerdeployments/build-general-linux-dev
status:
  availableReplicas: 4
  desiredReplicas: 4
  readyReplicas: 4
  replicas: 4
  updatedReplicas: 4
spec:
  effectiveTime: '2023-02-13T22:38:06Z'
  replicas: 4
  template:
    spec:
      automountServiceAccountToken: false
      dockerEnabled: false
      dockerdWithinRunnerContainer: true
      enterprise: <org-name>
      env:
        - name: DISABLE_RUNNER_UPDATE
          value: 'true'
      group: build-general-linux-dev
      image: accnewzealand/actions-runner-general-build:0.0.11
      imagePullPolicy: IfNotPresent
      imagePullSecrets:
        - name: dockerconfigjson
      labels:
        - build-general-linux-dev
      nodeSelector:
        pool: workload

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  creationTimestamp: '2023-02-13T21:55:43Z'
  generation: 3
  managedFields:
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:maxReplicas: {}
          f:minReplicas: {}
          f:scaleDownDelaySecondsAfterScaleOut: {}
          f:scaleTargetRef:
            f:name: {}
          f:scaleUpTriggers: {}
      manager: Terraform
      operation: Apply
      time: '2023-02-13T21:55:43Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:desiredReplicas: {}
          f:lastSuccessfulScaleOutTime: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-02-13T22:38:55Z'
  name: build-general-linux-dev-autoscaler
  namespace: actions-runner-system-build
  resourceVersion: '13018955'
  uid: efac8478-6a55-4093-894a-9ec47bcb59d3
  selfLink: >-
    /apis/actions.summerwind.dev/v1alpha1/namespaces/actions-runner-system-build/horizontalrunnerautoscalers/build-general-linux-dev-autoscaler
status:
  desiredReplicas: 4
  lastSuccessfulScaleOutTime: '2023-02-13T22:38:06Z'
spec:
  maxReplicas: 8
  minReplicas: 4
  scaleDownDelaySecondsAfterScaleOut: 30
  scaleTargetRef:
    name: build-general-linux-dev
  scaleUpTriggers:
    - duration: 30m0s
      githubEvent:
        workflowJob: {}

To Reproduce

1. Port forward port 8443 from arc pod
2. View /metrics endpoint in browser
3. Look for new metrics mentioned in https://github.com/actions/actions-runner-controller/discussions/1713

Describe the bug

The new metrics described in https://github.com/actions/actions-runner-controller/discussions/1713 are not in the arc, or metrics-server pod. The old one's are still there.

Pods:

NAME                                        READY   STATUS    RESTARTS        AGE
build-arc-bc4f7cfff-f7hsr                   1/1     Running   0          105s
build-arc-metrics-server-6c4c8747d-zt6g9    1/1     Running   0          33s
build-arc-webhook-server-84d44644d5-4pnd8   1/1     Running   0          26s
build-general-linux-dev-2qrk2-fgmrz         1/1     Running   0          5d21h
build-general-linux-dev-2qrk2-tbq2v         1/1     Running   0          5d22h
build-general-linux-dev-2qrk2-vjzwt         1/1     Running   0          5d21h
build-general-linux-dev-2qrk2-x6p82         1/1     Running   0          5d21h

Curl endpoint:

runner@build-general-linux-dev-2qrk2-tbq2v:/$ curl -k http://172.25.53.8:8443/metrics | grep horizontalrunnerautoscaler_
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

# HELP horizontalrunnerautoscaler_spec_max_replicas maxReplicas of HorizontalRunnerAutoscaler
# TYPE horizontalrunnerautoscaler_spec_max_replicas gauge
horizontalrunnerautoscaler_spec_max_replicas{horizontalrunnerautoscaler="build-general-linux-dev-autoscaler",namespace="actions-runner-system-build"} 8
100# HELP horizontalrunnerautoscaler_spec_min_replicas minReplicas of HorizontalRunnerAutoscaler
 6# TYPE horizontalrunnerautoscaler_spec_min_replicas gauge
587horizontalrunnerautoscaler_spec_min_replicas{horizontalrunnerautoscaler="build-general-linux-dev-autoscaler",namespace="actions-runner-system-build"} 4
5    0 65875    0     0  27.4M      0 --:--:-- --:--:-- --:--:-- 31.4M

Describe the expected behavior

The new metrics should be shown, instead the old one's are. I have tried this again by removing the Helm deployment, manually removing all the CRD's and then deploying chart 0.22.0 again, still the new metrics are not being shown.

Whole Controller Logs

https://gist.github.com/philthynz/49cdd50ebcae5ecca591b69dd832d8a5

Whole Runner Pod Logs

https://gist.github.com/philthynz/20c0625e1c0dabc2a360528aecdb302d

Additional Context

No response

Start with the controller and metrics-server /metrics endpoints, then compare their output with the metrics described in discussion 1713. Check how the reported controller version 0.27.0 and Helm chart 0.22.0 expose those metrics. Done means the new metrics appear instead of only the older metrics.