Runner hangs in job cancellation redelivery loop on v2.336.0 and never exits (#4312 not fully fixed)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
On v2.336.0, an ephemeral runner (ARC gha-runner-scale-set, dind mode) received a job cancellation and then hung indefinitely in a cancellation redelivery loop: the JobDispatcher logged Job cancellation request <id> received, cancellation timeout 5 minutes. several times per second, forever. The runner never completed the cancellation and never exited. GitHub deregistered it server side after the timeout, leaving a pod that Kubernetes considers healthy but that GitHub no longer counts as a registered runner.
This is the same failure mode as #4312, which was closed as completed on 2026-07-15, presumably on the basis of the v2.336.0 changes ("Exit ephemeral runners on broker acknowledge job-not-found" #4540, "Wait for worker to finish during cancel"). Our incident occurred on v2.336.0, roughly six hours after rolling it out, so those changes do not fully cover this path. Related: actions/actions-runner-controller#4307, actions/actions-runner-controller#4155.
To Reproduce
Not deterministic; observed occurrence:
- Ephemeral runner (ARC scale set, dind) picks up a job that runs containers via the dind sidecar
- The workflow run is cancelled while that job is executing (fail-fast from a matrix job failure)
- Runner receives the cancellation but never finishes it; JobDispatcher loops on the redelivered cancellation message indefinitely
Expected behavior
After the 5 minute cancellation timeout, the runner force-terminates the worker and exits (it is ephemeral, and its registration is gone), so the controller can replace it.
Runner Version and Platform
v2.336.0, Linux x64 container (ARC gha-runner-scale-set 0.12.0 on GKE, dind mode).
What's not working?
Timeline (2026-07-22, UTC):
- 11:24 job starts on the ephemeral runner
- 11:29 the workflow run is cancelled
- 11:33:58 runner logs
Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes. - The same line then repeats several times per second, indefinitely (still looping 40+ minutes later when we intervened)
- End state: pod Running 2/2 in Kubernetes, runner absent from GitHub. Scale set stats:
totalAssignedJobs: 1, totalRegisteredRunners: 0. The listener counted the zombie towards its target, so the next queued job waited ~25 minutes until we manually deleted the EphemeralRunner
Job Log Output
The cancelled job shows as cancelled in the UI; the hang is runner side only.
Runner and Worker's Diagnostic Logs
Runner container log excerpt (identical lines repeating at ~3 Hz until manual pod deletion):
[RUNNER 2026-07-22 11:33:58Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
[RUNNER 2026-07-22 11:33:58Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
[RUNNER 2026-07-22 11:33:59Z INFO JobDispatcher] Job cancellation request d6802bf9-7688-5b26-acc6-8d3f9e224022 received, cancellation timeout 5 minutes.
...
Happy to provide fuller diagnostic logs on request.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the JobDispatcher cancellation path and compare the repeated cancellation logs with the v2.336.0 changes referenced in #4540 and #4312. Reproduce cancellation of a container-running job on an ephemeral Linux runner, then verify that the worker is force-terminated after five minutes and the runner exits so it can be replaced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100