Pods stuck in Terminating state

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

summerwind/actions-runner-controller:v0.27.3

Helm Chart Version

actions-runner-controller-0.23.2

CertManager Version

v1.8.0

Deployment Method

Helm

cert-manager installation

Yes

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

apiVersion: v1
kind: Pod
metadata:
  annotations:
    actions-runner-controller/token-expires-at: "2023-10-17T17:42:53Z"
    actions-runner/github-api-creds-secret: softrams-github-secret
    actions-runner/id: "34087"
    actions-runner/runner-completion-wait-start-timestamp: "2023-10-17T17:02:42Z"
    actions-runner/unregistration-failure-message: Bad request - Runner "softrams-2m2v8-2m456"
      is still running a job"
    actions-runner/unregistration-start-timestamp: "2023-10-17T17:02:42Z"
    kubernetes.io/psp: eks.privileged
    sync-time: "2023-10-17T16:42:53Z"
  creationTimestamp: "2023-10-17T16:42:53Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-10-17T17:02:39Z"
  finalizers:
  - actions.summerwind.dev/runner-pod
  labels:
    actions-runner: ""
    actions-runner-controller/inject-registration-token: "true"
    pod-template-hash: 684c9c4dcf
    runner-deployment-name: softrams
    runner-template-hash: 695ddbd496
  name: softrams-2m2v8-2m456
  namespace: actions-runner-system
  ownerReferences:
  - apiVersion: actions.summerwind.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Runner
    name: softrams-2m2v8-2m456
    uid: b35ffec9-0539-491f-a290-6c3ea16556aa
  resourceVersion: "227053726"
  uid: ace9801d-458f-4f71-b768-c71176fd56fd
spec:
  containers:
  - env:
    - name: RUNNER_ORG
      value: softrams
    - name: RUNNER_REPO
    - name: RUNNER_ENTERPRISE
    - name: RUNNER_LABELS
      value: self-hosted,linux,ubuntu-latest,ubuntu-18.04,ubuntu-20.04
    - name: RUNNER_GROUP
    - name: DOCKER_ENABLED
      value: "true"
    - name: DOCKERD_IN_RUNNER
      value: "false"
    - name: GITHUB_URL
      value: https://github.com/
    - name: RUNNER_WORKDIR
      value: /runner/_work
    - name: RUNNER_EPHEMERAL
      value: "true"
    - name: RUNNER_STATUS_UPDATE_HOOK
      value: "false"
    - name: GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT
      value: actions-runner-controller/v0.27.3
    - name: DOCKER_HOST
      value: unix:///run/docker/docker.sock
    - name: RUNNER_NAME
      value: softrams-2m2v8-2m456
    - name: RUNNER_TOKEN
      value: A2IJC4U7WPPUATJAJ7L77FTFF3DZ3AVPNFXHG5DBNRWGC5DJN5XF62LEZYA2PBTSWFUW443UMFWGYYLUNFXW4X3UPFYGLN2JNZ2GKZ3SMF2GS33OJFXHG5DBNRWGC5DJN5XA
    image: summerwind/actions-runner:v2.303.0-ubuntu-22.04
    imagePullPolicy: Always
    name: runner
    resources: {}
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /runner
      name: runner
    - mountPath: /runner/_work
      name: work
    - mountPath: /run/docker
      name: docker-sock
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzwml
      readOnly: true
  - args:
    - dockerd
    - --host=unix:///run/docker/docker.sock
    - --group=$(DOCKER_GROUP_GID)
    - --registry-mirror=http://docker-registry.docker-registry:5000
    env:
    - name: DOCKER_GROUP_GID
      value: "121"
    image: docker:23.0.5-dind
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - timeout "${RUNNER_GRACEFUL_STOP_TIMEOUT:-15}" /bin/sh -c "echo 'Prestop
            hook started'; while [ -f /runner/.runner ]; do sleep 1; done; echo 'Waiting
            for dockerd to start'; while ! pgrep -x dockerd; do sleep 1; done; echo
            'Prestop hook stopped'" >/proc/1/fd/1 2>&1
    name: docker
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /runner
      name: runner
    - mountPath: /run/docker
      name: docker-sock
    - mountPath: /runner/_work
      name: work
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-nzwml
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: regcred
  nodeName: ip-172-16-2-67.ec2.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: runner
  - emptyDir: {}
    name: work
  - emptyDir:
      medium: Memory
      sizeLimit: 1M
    name: docker-sock
  - name: kube-api-access-nzwml
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-10-17T16:42:53Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-10-17T16:43:09Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-10-17T16:43:09Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-10-17T16:42:53Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6b20ac8c94f013a8820d46e5391729ff8206731aaf5d60c90fd344d3f20cc4ff
    image: docker.io/library/docker:23.0.5-dind
    imageID: docker.io/library/docker@sha256:f23f0a4013f184f6af3a3892dd12eba74bdbc5988d2a54ae468a8a6a44078434
    lastState: {}
    name: docker
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-10-17T16:43:08Z"
  - containerID: containerd://3a42625e2e9dd94b0c5a3f597bee4a953c37382e1601a6fa3332d04bf4cb90ac
    image: docker.io/summerwind/actions-runner:v2.303.0-ubuntu-22.04
    imageID: docker.io/summerwind/actions-runner@sha256:90bac4a220c9a5b501d822a1e59b22f842a32c4e4d72b82ff9ea955a2ef7fbe2
    lastState: {}
    name: runner
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-10-17T16:43:07Z"
  hostIP: 172.16.2.67
  phase: Running
  podIP: 172.16.2.72
  podIPs:
  - ip: 172.16.2.72
  qosClass: BestEffort
  startTime: "2023-10-17T16:42:53Z"

To Reproduce

Terminate a node that is running the pod.

Describe the bug

When the node gets removed/terminated, the runner pod is stuck in Terminating state. The restartpolicy for the pod is set to Never. Observed that each of the containers inside the pod is currently in the running state while the node has been removed completely. So, the finalizer assumes the container is still running and not removing the runner pod which is stuck in Terminating state.

Describe the expected behavior

The pods that are stuck in terminating state must be removed gracefully.

Whole Controller Logs

2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-9t6zs"}
2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-q2t9w"}
2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-zkcp5"}
2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-75r9n"}
2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-8mgzs"}
2023-10-20T12:04:13Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-r446l"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-9fbpz"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-fmfcf"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-8l7cn"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-ptdd9"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-j8v8s"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-f9lpf"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-6l8bc"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-rz595"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-plq4r"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-qf4hd"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-9t6zs"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-9l58l"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-2m456"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-znjzc"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-lvfvk"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-q2t9w"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-zkcp5"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-75r9n"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-8mgzs"}
2023-10-20T12:04:14Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-r446l"}
2023-10-20T12:04:15Z    INFO    runnerpod       Runner pod is annotated to wait for completion, and the runner container is not restarting      {"runnerpod": "actions-runner-system/softrams-2m2v8-8l7cn"}

Whole Runner Pod Logs

Defaulted container "runner" out of: runner, docker
Error from server (NotFound): pods "ip-172-16-2-67.ec2.internal" not found

Additional Context

No response

Start with the runnerpod reconciliation and finalizer handling, using the reported runnerpod logs and Pod manifest as the reproduction evidence. Reproduce the issue by terminating a node that runs the Pod, then verify that a Pod with the runner-pod finalizer is removed gracefully rather than remaining in Terminating state.