Kubernetes mode terminating early or not terminating at all.

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Create a workflow with a job that sleeps for 7 minutes and uses a ARC runner scale set configured in kubernetes mode.

Run the workflow.

Describe the bug

Even if the workflow should have slept for 7 minutes, it completes successfully after 4 minutes OR it never completes.

Terminating Early:

Never Terminating:

Describe the expected behavior

The workflow should have completed only after the sleep command completed.

Additional Context

values:
  - containerMode:
      kubernetesModeWorkVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: gh-runner-class
      type: kubernetes
    controllerServiceAccount:
      name: arc-gha-rs-controller
      namespace: arc-systems
    githubConfigSecret: sandbox
    githubConfigUrl: https://github.com/****
    githubServerTLS:
      certificateFrom:
        configMapKeyRef:
          key: ca.crt
          name: crt
      runnerMountPath: /usr/local/share/ca-certificates/
    maxRunners: 200
    minRunners: 5
    proxy:
      http:
        url: '*******'
      https:
        url: '*******'
      noProxy:
      - '*******'
      - '*******'
      - '*******'
      - '*******'
      - '*******'
    runnerGroup: preprod
    runnerScaleSetName: sandbox
    template:
      spec:
        containers:
        - command:
          - /home/runner/run.sh
          env:
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
          image: ghcr.io/actions/actions-runner:latest
          name: runner
        securityContext:
          fsGroup: 123

Controller Logs

[actions-runner-controller logs](https://gist.github.com/Josh-Engle/ea789d7eae1ca30e472fd90fba86bece)

Runner Pod Logs

[Runner Pod Logs](https://gist.github.com/Josh-Engle/86fcca504a8c3175e87b1a1efbd3f794)

### kubectl describe output on job that never terminated.

Name:             ****************.k8s-qrp6d-runner-9f8xw
Namespace:        arc-runners
Priority:         0
Service Account:  *************.k8s-gha-rs-kube-mode
Node:             aks-runnerpool-35648369-vmss000015/10.139.31.13
Start Time:       Wed, 05 Jun 2024 11:48:26 -0700
Labels:           actions-ephemeral-runner=True
                  actions.github.com/organization=********************
                  actions.github.com/scale-set-name=****************
                  actions.github.com/scale-set-namespace=arc-runners
                  app.kubernetes.io/component=runner
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.9.0
                  pod-template-hash=67bb8777b7
Annotations:      actions.github.com/patch-id: 70
                  actions.github.com/runner-group-name: *****************
                  actions.github.com/runner-scale-set-name: ******************
                  actions.github.com/runner-spec-hash: cdff6bcb
Status:           Running
IP:               10.244.0.76
IPs:
  IP:           10.244.0.76
Controlled By:  EphemeralRunner/*******************.k8s-qrp6d-runner-9f8xw
Containers:
  runner:
    Container ID:  containerd://71795364d03d6e29cc0a7d40623120040354ab416e7139528b39841c164d47f1
    Image:         ghcr.io/actions/actions-runner:latest
    Image ID:      ghcr.io/actions/actions-runner@sha256:95db6fbb020b9f734e8a00389291dae766f0e6ad3d1171ae2d68e9ad8ac4a985
    Port:          <none>
    Host Port:     <none>
    Command:
      /home/runner/run.sh
    State:          Running
      Started:      Wed, 05 Jun 2024 11:48:50 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER:    false
      ACTIONS_RUNNER_CONTAINER_HOOKS:          /home/runner/k8s/index.js
      ACTIONS_RUNNER_POD_NAME:                 *********************.k8s-qrp6d-runner-9f8xw (v1:metadata.name)
      NODE_EXTRA_CA_CERTS:                     /usr/local/share/ca-certificates/ca.crt
      RUNNER_UPDATE_CA_CERTS:                  1
      ACTIONS_RUNNER_INPUT_JITCONFIG:          <set to the key 'jitToken' in secret '*************.k8s-qrp6d-runner-9f8xw'>  Optional: false
      GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT:  actions-runner-controller/0.9.0
      http_proxy:                              <set to the key 'http_proxy' in secret '*************.k8s-qrp6d-754b578d-runner-proxy'>   Optional: false
      https_proxy:                             <set to the key 'https_proxy' in secret '*************.k8s-qrp6d-754b578d-runner-proxy'>  Optional: false
      no_proxy:                                <set to the key 'no_proxy' in secret '*************.k8s-qrp6d-754b578d-runner-proxy'>     Optional: false
    Mounts:
      /home/runner/_work from work (rw)
      /usr/local/share/ca-certificates/ca.crt from github-server-tls-cert (rw,path="ca.crt")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c6wc6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  github-server-tls-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      crt
    Optional:  false
  work:
    Type:          EphemeralVolume (an inline specification for a volume that gets created and deleted with the pod)
    StorageClass:  gh-runner-class
    Volume:        
    Labels:            <none>
    Annotations:       <none>
    Capacity:      
    Access Modes:  
    VolumeMode:    Filesystem
  kube-api-access-c6wc6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Reproduce the 7-minute sleep workflow with the Kubernetes-mode runner scale set, then inspect /home/runner/k8s/index.js alongside the linked controller and runner logs. Compare the early-termination and never-terminating cases, and verify that the workflow remains running until the sleep command finishes.