Pods created by `RunnerSet` are terminated while still running jobs on a rolling update

Controller Version

0.24.0

Helm Chart Version

0.19.0

CertManager Version

1.4.1

Deployment Method

Helm

cert-manager installation

Deploying cert-manager Helm chart from https://charts.jetstack.io/

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: clr-runner
  namespace: actions-runner-groups
spec:
  dockerdWithinRunnerContainer: false
  ephemeral: true
  labels:
  - clr-runner
  organization: color
  replicas: 1
  selector:
    matchLabels:
      app: clr-runner
  serviceName: clr-runner
  template:
    metadata:
      labels:
        app: clr-runner
    spec:
      containers:
      - image: 301643779712.dkr.ecr.us-east-1.amazonaws.com/color-actions-runner:master_c704c14c
        name: runner
        resources:
          limits:
            cpu: 8
            memory: 32Gi
          requests:
            cpu: 8
            memory: 32Gi
      - image: public.ecr.aws/docker/library/docker:dind
        name: docker
      securityContext:
        fsGroup: 1000
      serviceAccountName: actions-runner
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: clr-runner
  namespace: actions-runner-groups
spec:
  maxReplicas: 30
  metrics:
  - scaleDownFactor: "0.7"
    scaleDownThreshold: "0.2"
    scaleUpFactor: "2.5"
    scaleUpThreshold: "0.5"
    type: PercentageRunnersBusy
  minReplicas: 1
  scaleDownDelaySecondsAfterScaleOut: 3600
  scaleTargetRef:
    kind: RunnerSet
    name: clr-runner

To Reproduce

1. Change the `image` for the `runner` container in the manifest above
2. Launch a workflow that starts running some jobs on the runners
3. `kubectl apply` it to update the runners

Describe the bug

All runner pods are restarted within < a minute of the kubectl apply, even those that are still running jobs. The in-flight jobs are dropped and appear hung in the GitHub UI (they eventually time out).

Describe the expected behavior

Pods aren't terminated until they've finished running in-flight jobs

Controller Logs

Will post in separate comment to avoid #1533

Runner Pod Logs

N/A pods are deleted

Additional Context

No response

Start by reproducing the rolling update with the supplied RunnerSet and HorizontalRunnerAutoscaler manifests, then inspect the controller logs and RunnerSet reconciliation behavior. The fix is done when updating the runner image does not terminate pods with in-flight jobs, while completed jobs and normal rolling updates still work.