actions / actions/runner

Runner.Worker hangs at 100% CPU in "Finalize job" on macOS — regression in 2.336.0

Open
#4,575 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

On macOS (Apple Silicon), Runner.Worker hangs forever during Finalize job, spinning at 100% CPU on one core. The job's steps all succeed; the job never reports completion and is eventually cancelled by timeout-minutes. The timeout ends the job on the GitHub side but does not kill the local process, so orphaned workers accumulate — I found 4 alive simultaneously, the oldest at ~2 days, each pegging a core (load average 5.06 on an otherwise idle machine).

The hang is in the post-job process scan. The worker log ends exactly here and then repeats the same HostContext lines every 10 seconds indefinitely:

18:17:55Z [JobRunner] Finalize job.
18:17:55Z [JobExtension] Total accessible running process: 357.
18:17:55Z [JobExtension] Inspecting process environment variables. PID: 3666 (sleep)
18:17:55Z [LinuxProcessExtensions] Read env from output of `ps e -p 3666 -o command`
   ... never advances; log grows forever

It wedges on the first process inspected. By the time I investigated, PID 3666 (a sleep) no longer existed, and there was a <defunct> ps process on the box. So the sequence appears to be:

  1. The scan snapshots the process list.
  2. For each PID it spawns ps e -p <pid> -o command to read the env.
  3. The target process exits during that read.
  4. The ps child is reaped into a zombie and the invoker's wait never completes.

That makes it a race, which matches the intermittency — it does not happen on every job.

sample on the stuck process shows a deep, repeating managed stack with no blocking syscall, consistent with a spin rather than a blocked wait.

Regression between 2.335.1 and 2.336.0

The runner self-updated from 2.335.1 to 2.336.0 on 2026-07-23. I classified all 83 worker logs on the machine by whether they end in the repeating loop:

Runner version Jobs Hung
2.335.1 65 0
2.336.0 18 10 (56%)

Pinning back to 2.335.1 with --disableupdate resolved it: the next deploy finished in 45s with Complete job: success and [Worker] Job completed. in the log — a line absent since the update.

To Reproduce

Not deterministic (it's a race), but the conditions here were:

  1. Self-hosted runner on macOS 26.3.1, Apple Silicon, runner 2.336.0, running as a LaunchAgent service.
  2. A job whose last step spawns short-lived child processes (a shell until/health-check loop using sleep) and a machine with a few hundred processes running (357 at the time).
  3. Let the job finish. Occasionally the worker wedges at "Finalize job" instead of completing.
Expected behavior

The process-environment scan should tolerate a PID that disappears mid-inspection — skip it and continue — rather than wedging the worker permanently. Ideally the scan is also bounded by a timeout so a single unreadable process can't hang job finalization.

Runner Version and Platform

Version: 2.336.0 (hangs) / 2.335.1 (does not)
OS: macOS 26.3.1, Apple Silicon (ARM64)

What's not working?

Self-hosted runner on macOS.

Job Log Output
18:17:55Z INFO ProcessInvokerWrapper] Finished process 3446 with exit code 0, and elapsed time 00:01:26.1561025.
18:17:55Z INFO StepsRunner] Step result: succeeded
18:17:55Z INFO JobRunner] Finalize job.
18:17:55Z INFO JobExtension] Initialize Env context
18:17:55Z INFO JobExtension] Initialize steps context
18:17:55Z INFO JobExtension] Total accessible running process: 357.
18:17:55Z INFO JobExtension] Inspecting process environment variables. PID: 3666 (sleep)
18:17:55Z INFO LinuxProcessExtensions] Read env from output of `ps e -p 3666 -o command`
18:17:55Z INFO ProcessInvokerWrapper] Starting process:
18:17:55Z INFO ProcessInvokerWrapper]   File name: 'ps'
18:17:55Z INFO ProcessInvokerWrapper]   Arguments: 'e -p 3666 -o command'
18:17:55Z INFO ProcessInvokerWrapper]   Working directory: '/Users/<user>/actions-runner'
18:17:55Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[from here on, only this triplet repeats every 10 seconds, forever:]
18:44:49Z INFO HostContext] Well known directory 'Bin': '/Users/<user>/actions-runner/bin.2.336.0'
18:44:49Z INFO HostContext] Well known directory 'Root': '/Users/<user>/actions-runner'
18:44:49Z INFO HostContext] Well known directory 'Work': '/Users/<user>/actions-runner/_work'

Process state while hung (note PPID 1 and 100% CPU):

  PID  PPID STAT  %CPU     ELAPSED COMMAND
 3724     1 R    100.0       34:06 Runner.Worker spawnclient 157 160
 4776     1 R    100.0 01-22:40:20 Runner.Worker spawnclient 155 158
66785     1 R    100.0    04:41:52 Runner.Worker spawnclient 157 160
99550     1 R    100.0 01-23:03:53 Runner.Worker spawnclient 155 158
Runner and Worker's Diagnostic Logs

Relevant excerpts included above. Happy to attach full _diag logs if useful.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Finalize job path from the JobExtension logs through LinuxProcessExtensions and ProcessInvokerWrapper, focusing on the ps e -p <pid> -o command invocation. Reproduce the race on macOS 26.3.1 with runner 2.336.0 and inspect the diagnostic logs and process state. Done means a disappearing PID no longer leaves Runner.Worker spinning, and the process scan lets job finalization complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, macos
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.