Runners recreated indefinitely

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.4

Helm Chart Version

0.23.3

CertManager Version

1.12.1

Deployment Method

Helm

cert-manager installation

Innstalled cert-manager from official helm chart

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: eag-runners-test
  namespace: github-actions
spec:
  #replicas: 2
  template:
    spec:
      image: registry_url/github-runner-ubuntu:latest
      imagePullPolicy: Always
      labels:
        - runner-eag
      repository: my-org/eag-runners-test
      githubAPICredentialsFrom:
        secretRef:
          name: controller-manager-xxx
      ephemeral: true
      effectiveTime:
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: eag-runners-test-autoscaler
  namespace: github-actions
spec:
  scaleDownDelaySecondsAfterScaleOut: 100
  scaleTargetRef:
    kind: RunnerDeployment
    name: eag-runners-test
  minReplicas: 1
  maxReplicas: 3
  scaleUpTriggers:
    - githubEvent:
        workflowJob: { }
  metrics:
    - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
      repositoryNames:
        - my-org/eag-runners-test

To Reproduce

1. deploy ARC with helm in a kubernetes cluster
2. configure a Runner Deployment and HRA in the cluster (with argo)
3. use the custom label to trigger the actions in the k8s cluster pods
4. Wait 20 minutes from the last trigger and more pods than expected appears indefinitely

Describe the bug

The runners are created indefinitely after you run several actions and no have activity, even than you have defined the HRA to have minimum replicas of 1.

Describe the expected behavior

The runners should scale back to 1.

Whole Controller Logs

https://gist.github.com/ealogar/8046605ae7297a08accddb97ac1e2a69

Whole Runner Pod Logs

https://gist.github.com/ealogar/153cf774db73c6ff1596278e3d3c2963

Additional Context

It's strange that the controller is trying to update labels of the running pods.

Start by reproducing the RunnerDeployment and HorizontalRunnerAutoscaler configuration from the issue with ARC 0.27.4 and Helm chart 0.23.3. Review the linked whole controller logs and runner pod logs, focusing on repeated runner creation and label updates after activity stops. Done means runners scale back to the configured minimum of one without indefinite recreation.