actions / actions/runner

Runner hangs in job cancellation redelivery loop on v2.336.0 and never exits (#4312 not fully fixed)

Open
#4,569 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

On v2.336.0, an ephemeral runner (ARC gha-runner-scale-set, dind mode) received a job cancellation and then hung indefinitely in a cancellation redelivery loop: the JobDispatcher logged Job cancellation request <id> received, cancellation timeout 5 minutes. several times per second, forever. The runner never completed the cancellation and never exited. GitHub deregistered it server side after the timeout, leaving a pod that Kubernetes considers healthy but that GitHub no longer counts as a registered runner.

This is the same failure mode as #4312, which was closed as completed on 2026-07-15, presumably on the basis of the v2.336.0 changes ("Exit ephemeral runners on broker acknowledge job-not-found" #4540, "Wait for worker to finish during cancel"). Our incident occurred on v2.336.0, roughly six hours after rolling it out, so those changes do not fully cover this path. Related: actions/actions-runner-controller#4307, actions/actions-runner-controller#4155.

To Reproduce

Not deterministic; observed occurrence:

  1. Ephemeral runner (ARC scale set, dind) picks up a job that runs containers via the dind sidecar
  2. The workflow run is cancelled while that job is executing (fail-fast from a matrix job failure)
  3. Runner receives the cancellation but never finishes it; JobDispatcher loops on the redelivered cancellation message indefinitely

Expected behavior

After the 5 minute cancellation timeout, the runner force-terminates the worker and exits (it is ephemeral, and its registration is gone), so the controller can replace it.

Runner Version and Platform

v2.336.0, Linux x64 container (ARC gha-runner-scale-set 0.12.0 on GKE, dind mode).

What's not working?

Timeline (2026-07-22, UTC):

  • 11:24 job starts on the ephemeral runner
  • 11:29 the workflow run is cancelled
  • 11:33:58 runner logs Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
  • The same line then repeats several times per second, indefinitely (still looping 40+ minutes later when we intervened)
  • End state: pod Running 2/2 in Kubernetes, runner absent from GitHub. Scale set stats: totalAssignedJobs: 1, totalRegisteredRunners: 0. The listener counted the zombie towards its target, so the next queued job waited ~25 minutes until we manually deleted the EphemeralRunner

Job Log Output

The cancelled job shows as cancelled in the UI; the hang is runner side only.

Runner and Worker's Diagnostic Logs

Runner container log excerpt (identical lines repeating at ~3 Hz until manual pod deletion):

[RUNNER 2026-07-22 11:33:58Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
[RUNNER 2026-07-22 11:33:58Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
[RUNNER 2026-07-22 11:33:59Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
...

Happy to provide fuller diagnostic logs on request.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the JobDispatcher cancellation path and compare the repeated cancellation logs with the v2.336.0 changes referenced in #4540 and #4312. Reproduce cancellation of a container-running job on an ephemeral Linux runner, then verify that the worker is force-terminated after five minutes and the runner exits so it can be replaced.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.