Runner.Worker hangs at 100% CPU in "Finalize job" on macOS — regression in 2.336.0
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
On macOS (Apple Silicon), Runner.Worker hangs forever during Finalize job, spinning at 100% CPU on one core. The job's steps all succeed; the job never reports completion and is eventually cancelled by timeout-minutes. The timeout ends the job on the GitHub side but does not kill the local process, so orphaned workers accumulate — I found 4 alive simultaneously, the oldest at ~2 days, each pegging a core (load average 5.06 on an otherwise idle machine).
The hang is in the post-job process scan. The worker log ends exactly here and then repeats the same HostContext lines every 10 seconds indefinitely:
18:17:55Z [JobRunner] Finalize job.
18:17:55Z [JobExtension] Total accessible running process: 357.
18:17:55Z [JobExtension] Inspecting process environment variables. PID: 3666 (sleep)
18:17:55Z [LinuxProcessExtensions] Read env from output of `ps e -p 3666 -o command`
... never advances; log grows forever
It wedges on the first process inspected. By the time I investigated, PID 3666 (a sleep) no longer existed, and there was a <defunct> ps process on the box. So the sequence appears to be:
- The scan snapshots the process list.
- For each PID it spawns
ps e -p <pid> -o commandto read the env. - The target process exits during that read.
- The
pschild is reaped into a zombie and the invoker's wait never completes.
That makes it a race, which matches the intermittency — it does not happen on every job.
sample on the stuck process shows a deep, repeating managed stack with no blocking syscall, consistent with a spin rather than a blocked wait.
Regression between 2.335.1 and 2.336.0
The runner self-updated from 2.335.1 to 2.336.0 on 2026-07-23. I classified all 83 worker logs on the machine by whether they end in the repeating loop:
| Runner version | Jobs | Hung |
|---|---|---|
| 2.335.1 | 65 | 0 |
| 2.336.0 | 18 | 10 (56%) |
Pinning back to 2.335.1 with --disableupdate resolved it: the next deploy finished in 45s with Complete job: success and [Worker] Job completed. in the log — a line absent since the update.
To Reproduce
Not deterministic (it's a race), but the conditions here were:
- Self-hosted runner on macOS 26.3.1, Apple Silicon, runner 2.336.0, running as a LaunchAgent service.
- A job whose last step spawns short-lived child processes (a shell
until/health-check loop usingsleep) and a machine with a few hundred processes running (357 at the time). - Let the job finish. Occasionally the worker wedges at "Finalize job" instead of completing.
Expected behavior
The process-environment scan should tolerate a PID that disappears mid-inspection — skip it and continue — rather than wedging the worker permanently. Ideally the scan is also bounded by a timeout so a single unreadable process can't hang job finalization.
Runner Version and Platform
Version: 2.336.0 (hangs) / 2.335.1 (does not)
OS: macOS 26.3.1, Apple Silicon (ARM64)
What's not working?
Self-hosted runner on macOS.
Job Log Output
18:17:55Z INFO ProcessInvokerWrapper] Finished process 3446 with exit code 0, and elapsed time 00:01:26.1561025.
18:17:55Z INFO StepsRunner] Step result: succeeded
18:17:55Z INFO JobRunner] Finalize job.
18:17:55Z INFO JobExtension] Initialize Env context
18:17:55Z INFO JobExtension] Initialize steps context
18:17:55Z INFO JobExtension] Total accessible running process: 357.
18:17:55Z INFO JobExtension] Inspecting process environment variables. PID: 3666 (sleep)
18:17:55Z INFO LinuxProcessExtensions] Read env from output of `ps e -p 3666 -o command`
18:17:55Z INFO ProcessInvokerWrapper] Starting process:
18:17:55Z INFO ProcessInvokerWrapper] File name: 'ps'
18:17:55Z INFO ProcessInvokerWrapper] Arguments: 'e -p 3666 -o command'
18:17:55Z INFO ProcessInvokerWrapper] Working directory: '/Users/<user>/actions-runner'
18:17:55Z INFO ProcessInvokerWrapper] Require exit code zero: 'False'
[from here on, only this triplet repeats every 10 seconds, forever:]
18:44:49Z INFO HostContext] Well known directory 'Bin': '/Users/<user>/actions-runner/bin.2.336.0'
18:44:49Z INFO HostContext] Well known directory 'Root': '/Users/<user>/actions-runner'
18:44:49Z INFO HostContext] Well known directory 'Work': '/Users/<user>/actions-runner/_work'
Process state while hung (note PPID 1 and 100% CPU):
PID PPID STAT %CPU ELAPSED COMMAND
3724 1 R 100.0 34:06 Runner.Worker spawnclient 157 160
4776 1 R 100.0 01-22:40:20 Runner.Worker spawnclient 155 158
66785 1 R 100.0 04:41:52 Runner.Worker spawnclient 157 160
99550 1 R 100.0 01-23:03:53 Runner.Worker spawnclient 155 158
Runner and Worker's Diagnostic Logs
Relevant excerpts included above. Happy to attach full _diag logs if useful.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Finalize job path from the JobExtension logs through LinuxProcessExtensions and ProcessInvokerWrapper, focusing on the ps e -p <pid> -o command invocation. Reproduce the race on macOS 26.3.1 with runner 2.336.0 and inspect the diagnostic logs and process state. Done means a disappearing PID no longer leaves Runner.Worker spinning, and the process scan lets job finalization complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, macos
- Domain
- devtools, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100