runnerdeployment_spec_replicas metric does not update after replicas changed

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

v0.27.3

Helm Chart Version

0.23.2

CertManager Version

v1.11.0

Deployment Method

Helm

cert-manager installation

Cert-manager installed accordingly.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

-

To Reproduce

See description.

Describe the bug

While configuring Grafana dashboard and testing metrics I encountered strange bug.

runnerdeployment_spec_replicas gauge does not update after replica amount is changes.

Below you can find commands and outputs depicting this behavior:

1. Checking current runnerdeployment_spec_replicas amount and comparing to actual amount of pods.

$ curl -s localhost:8443/metrics | grep recog
runnerdeployment_spec_replicas{namespace="actions-runner-system",runnerdeployment="recogrollup-runnerdeploy"} 1

$ kubectl get runnerdeployments.actions.summerwind.dev -n actions-runner-system | grep recog
recogrollup-runnerdeploy                                           company-name/recog-rollup                     ["recogrollup-runner"]               1         1         1            1           12h

$ kubectl get pod -n actions-runner-system | grep recog
recogrollup-runnerdeploy-j2kw7-sp2xx               2/2     Running   0          12h

2. Deleting runnerdeployment manually and checking metric again (after 5 minutes). The value is still 1 however pod has been deleted

$ kubectl delete runnerdeployments.actions.summerwind.dev -n actions-runner-system recogrollup-runnerdeploy
runnerdeployment.actions.summerwind.dev "recogrollup-runnerdeploy" deleted

$ kubectl get pod -n actions-runner-system | grep recog

$ curl -s localhost:8443/metrics | grep recog
runnerdeployment_spec_replicas{namespace="actions-runner-system",runnerdeployment="recogrollup-runnerdeploy"} 1

After the pod has been deleted, runnerdeployment_spec_replicas still equals to 1.

Describe the expected behavior

Metric runnerdeployment_spec_replicas is updated with actual replicas amount.

Whole Controller Logs

2023-04-19T14:21:51Z    INFO    runner    Removed finalizer    {"runner": "actions-runner-system/recogrollup-runnerdeploy-j2kw7-sp2xx"}                                                                          
023-04-19T14:21:51Z    INFO    runnerpod    Runner has just been unregistered.    {"runnerpod": "actions-runner-system/recogrollup-runnerdeploy-j2kw7-sp2xx"}

Whole Runner Pod Logs

-

Additional Context

No response

Start by reproducing the report through the controller's localhost:8443/metrics endpoint while changing or deleting a RunnerDeployment, and compare the metric with kubectl output. Trace the metric's update and cleanup behavior in the controller and add or run coverage for replica changes and deletion. Done means runnerdeployment_spec_replicas no longer remains stale after the resource or its replicas change.