EphemeralRunner never fails runner pods stuck Pending on permanent image errors

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.14.2 (same code path on master @ 03328aa14f6435462f6a2add1f5ba79247da6e09)

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install gha-runner-scale-set with a runner template whose runner container image can never be pulled, e.g.
   `image: ghcr.io/actions/actions-runner:tag-that-does-not-exist`
   (any of these give the same result: an invalid reference such as `ghcr.io/actions/actions-runner::bad` -> InvalidImageName;
   `imagePullPolicy: Never` with the image absent on the node -> ErrImageNeverPull; a private image without a pull secret -> ImagePullBackOff).
2. Queue a workflow job targeting the scale set (or set minRunners: 1).
3. `kubectl get pods -n <runner ns>` shows the runner pod in `Pending` with STATUS `ImagePullBackOff` / `InvalidImageName` / `ErrImageNeverPull`.
4. Wait as long as you like (hours). Observe:
   - the pod stays Pending; kubelet retries the pull forever (ImagePullBackOff) or never retries (InvalidImageName, ErrImageNeverPull);
   - the EphemeralRunner stays in phase Pending/Running with `status.failures` empty;
   - the controller log for that EphemeralRunner only repeats "Runner container is still running; updating ephemeral runner status";
   - the queued job stays in "Waiting for a runner to come online"; the EphemeralRunnerSet counts the pod as pending capacity, so nothing else is scheduled for that slot.
5. Compare with a runner template whose *init container* fails: that pod is deleted, `status.failures` grows, and after maxFailures the EphemeralRunner is recreated / surfaced.

Describe the bug

The EphemeralRunner reconciler classifies the runner container purely by Terminated vs. not-Terminated. A pod whose runner container is stuck in Waiting because the image can never be pulled is indistinguishable, to the controller, from a healthy running runner:

• https://github.com/actions/actions-runner-controller/blob/03328aa14f6435462f6a2add1f5ba79247da6e09/controllers/actions.github.com/ephemeralrunner_controller.go#L332-L333 — cs := runnerContainerStatus(pod) then switch { ... }
• https://github.com/actions/actions-runner-controller/blob/03328aa14f6435462f6a2add1f5ba79247da6e09/controllers/actions.github.com/ephemeralrunner_controller.go#L379-L384 — case initContainerFailed(pod): is the only pre-Terminated failure that is handled (added in #4457).
• https://github.com/actions/actions-runner-controller/blob/03328aa14f6435462f6a2add1f5ba79247da6e09/controllers/actions.github.com/ephemeralrunner_controller.go#L386-L389 — case cs == nil: logs "Waiting for runner container status to be available" and returns.
• https://github.com/actions/actions-runner-controller/blob/03328aa14f6435462f6a2add1f5ba79247da6e09/controllers/actions.github.com/ephemeralrunner_controller.go#L391-L397 — case cs.State.Terminated == nil: logs "Runner container is still running" and returns.

cs.State.Waiting.Reason is never inspected anywhere in the controller (grep -n Waiting controllers/actions.github.com/ephemeralrunner_controller.go only hits log strings). For permanent reasons — InvalidImageName, ErrImageNeverPull, or an ImagePullBackOff whose message is manifest unknown / not found / auth denied — the pod never leaves Pending, so pod.Status.Phase == corev1.PodFailed is never reached either. deletePodAsFailed (https://github.com/actions/actions-runner-controller/blob/03328aa14f6435462f6a2add1f5ba79247da6e09/controllers/actions.github.com/ephemeralrunner_controller.go#L657-L690) is therefore never called, status.failures never grows, the maxFailures guard at L248 never fires, and the misconfiguration is invisible from the EphemeralRunner/AutoscalingRunnerSet objects. The stuck EphemeralRunner occupies a slot in the EphemeralRunnerSet's desired count, so the assigned job waits indefinitely for a runner that will never register.

This is the main-container analogue of #4457 (init container failure before PodFailed). The maintainer note on #4272 asked for retry-worthy cases to be filed individually — this is one.

Related: #4540 (open) adds a terminalContainerWaitingReasons map and routes such pods through deleteEphemeralRunnerOrPod as part of a larger "surface pod errors to the check run" feature. That PR would fix this, but it treats ImagePullBackOff as terminal immediately, which would kill large images that legitimately back off for a few minutes on cold nodes. Filing this as a standalone bug so the failure-accounting part can be fixed (and bounded) independently of the check-run annotation feature.

Describe the expected behavior

A runner pod that can never start should be handled like a failed pod: deleted via the existing deleteEphemeralRunnerOrPod -> deletePodAsFailed path so that

• status.failures grows and the pod is recreated with the existing backoff (failedRunnerBackoff),
• after maxFailures the EphemeralRunner is deleted/recreated (or marked Failed) with status.reason/status.message carrying the Waiting.Reason/Waiting.Message (e.g. InvalidImageName: couldn't parse image name ...),
• if a job is already assigned, the runner is removed from the service as deleteEphemeralRunnerOrPod already does.

Proposed fix (small, self-contained):

// controllers/actions.github.com/ephemeralrunner_controller.go

// Waiting reasons that kubelet will never recover from on its own.
var unrecoverableWaitingReasons = map[string]struct{}{
    "InvalidImageName":  {},
    "ErrImageNeverPull": {},
}

// Reasons kubelet keeps retrying; treat as failed only after a bounded grace period
// so large images on cold nodes are not killed prematurely.
var backoffWaitingReasons = map[string]struct{}{
    "ImagePullBackOff": {},
    "ErrImagePull":     {},
}

const imagePullGracePeriod = 10 * time.Minute

func runnerContainerImageUnrecoverable(pod *corev1.Pod, now time.Time) bool {
    cs := runnerContainerStatus(pod)
    if cs == nil || cs.State.Waiting == nil {
        return false
    }
    if _, ok := unrecoverableWaitingReasons[cs.State.Waiting.Reason]; ok {
        return true
    }
    if _, ok := backoffWaitingReasons[cs.State.Waiting.Reason]; ok {
        return !pod.CreationTimestamp.IsZero() && now.Sub(pod.CreationTimestamp.Time) > imagePullGracePeriod
    }
    return false
}

and, in the reconcile switch, before case cs == nil::

case runnerContainerImageUnrecoverable(pod, time.Now()):
    log.Info("Runner container image cannot be pulled, deleting pod as failed so it can be restarted",
        "reason", cs.State.Waiting.Reason, "message", cs.State.Waiting.Message)
    return ctrl.Result{}, r.deleteEphemeralRunnerOrPod(ctx, &ephemeralRunner, pod, log)

Because the pod object does not change while kubelet backs off, the ImagePullBackOff branch also needs return ctrl.Result{RequeueAfter: imagePullGracePeriod - age} from the Terminated == nil branch when the runner container is in a backoff reason and still young, otherwise the threshold is only re-evaluated on the next unrelated pod event. deletePodAsFailed should also copy cs.State.Waiting.Reason/Message into status.reason/message when pod.Status.Reason is empty (it is, for Pending pods), so the failure is readable from the EphemeralRunner.

Test plan: table-driven unit test for runnerContainerImageUnrecoverable (InvalidImageName -> true; ErrImageNeverPull -> true; ImagePullBackOff young -> false; ImagePullBackOff older than grace -> true; Running -> false; nil status -> false), plus an envtest case next to the existing init-container tests in ephemeralrunner_controller_test.go that patches ContainerStatuses[runner].State.Waiting.Reason = "InvalidImageName" and asserts the pod is deleted and Status.Failures has one entry. I am happy to send that PR if the approach is acceptable.

Additional Context

# gha-runner-scale-set values excerpt sufficient to reproduce
githubConfigUrl: https://github.com/<org>/<repo>
githubConfigSecret: <secret>
minRunners: 1
template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:tag-that-does-not-exist   # or ghcr.io/actions/actions-runner::bad
        command: ["/home/runner/run.sh"]

Controller Logs

# The only line the controller emits for the affected EphemeralRunner, repeated on every pod status event
# (ephemeralrunner_controller.go L392):
INFO EphemeralRunner Runner container is still running; updating ephemeral runner status {"ephemeralrunner": {"name":"<scale-set>-<hash>-runner-<id>","namespace":"<ns>"}}

# Never emitted for this pod:
#   "Deleting the ephemeral runner pod" / "Updating ephemeral runner status to track the failure count" (deletePodAsFailed)
#   "EphemeralRunner has failed more than 5 times" (maxFailures guard)

Runner Pod Logs

$ kubectl -n <ns> get pod <scale-set>-<hash>-runner-<id>
NAME                                   READY   STATUS             RESTARTS   AGE
<scale-set>-<hash>-runner-<id>         0/1     ImagePullBackOff   0          6h

$ kubectl -n <ns> describe pod <scale-set>-<hash>-runner-<id>
...
Containers:
  runner:
    State:          Waiting
      Reason:       ImagePullBackOff        # or InvalidImageName / ErrImageNeverPull
    Ready:          False
...
Events:
  Warning  Failed   ...  kubelet  Failed to pull image "ghcr.io/actions/actions-runner:tag-that-does-not-exist": ... manifest unknown
  Warning  Failed   ...  kubelet  Error: ErrImagePull
  Normal   BackOff  ...  kubelet  Back-off pulling image "ghcr.io/actions/actions-runner:tag-that-does-not-exist"
  Warning  Failed   ...  kubelet  Error: ImagePullBackOff

$ kubectl -n <ns> get ephemeralrunner <scale-set>-<hash>-runner-<id> -o jsonpath='{.status.failures}'
# (empty, indefinitely)

(No runner process ever starts, so there are no runner container logs.)

Analysis prepared with an AI agent operated by KR-Ravindra, who verified the code paths.

Start in controllers/actions.github.com/ephemeralrunner_controller.go at the EphemeralRunner reconcile switch, runnerContainerStatus handling, and deletePodAsFailed path; compare the existing init-container failure behavior. Read the related tests in ephemeralrunner_controller_test.go and run them first. Done means permanent image-pull failures are accounted for, bounded retries remain possible, failure details are visible in status, and the new unit and envtest cases pass.