actions / actions/runner

Self-hosted runner (macOS/ARM64): Runner.Worker deadlocks at job startup right after creating the results WebSocket (ingest.sock), holding the runner busy

Open
#4,558 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

On self-hosted macOS/ARM64 runners, Runner.Worker intermittently deadlocks at job startup, immediately after opening the results/live-log WebSocket. The worker process stays alive, keeps renewing the job lease (so GitHub shows the runner busy), but never spawns the first step — blocking all queued jobs for hours until the process is killed.

Runner Version and Platform
  • Runner: 2.335.1 (also reproduced on 2.334.0 — see below)
  • OS: macOS (Darwin 25.5.0), Apple Silicon (M4), self-hosted, running as a launchd service
  • Trigger: normal pull_request CI (matrix of Rust/gate jobs)
What's the expected behavior?

The worker sets up the results WebSocket and proceeds to run the job steps.

Symptom / evidence

The Worker_*.log freezes right after:

[ResultServer] Creating websocket client ...wss://results-receiver.actions.githubusercontent.com/_ws/ingest.sock
...
[JobServerQueue] Stop aggressive process web console line queue.

Then: no further log lines, no child process spawned (checkout never runs), ~0% CPU, for hours. JobDispatcher keeps logging Successfully renew job ..., so the runner reports busy=true to GitHub and no new jobs are dispatched to it.

sample <Runner.Worker pid> (macOS) shows all ~17 threads parked on .NET synchronization primitives and zero threads in socket/network I/O:

main thread: ... RunMain -> [managed] -> ObjectNative::WaitTimeout -> SyncBlock::Wait
             -> Thread::Block -> _pthread_cond_wait -> __psynch_cvwait
aggregate:   ~10x InternalWaitForMultipleObjectsEx, 4x WaitHandleNative::CorWaitOneNative,
             3x CorWaitOnePrioritizedNative, 2x ObjectNative::WaitTimeout - 0x SSLRead/recv

i.e. an internal deadlock, not a network stall (a network hang would show a thread blocked in a socket read).

Ruled out during investigation
  • Network: basic connectivity is fine; a curl WebSocket upgrade to results-receiver.../_ws/ingest.sock returns HTTP 401 in ~0.5s. Thread sample shows no thread doing network I/O.
  • Runner version: downgrading to 2.334.0 hangs identically at the same WebSocket point (both versions contain the same ResultServer/WebSocket code).
  • Thread-pool starvation: DOTNET_ThreadPool_ForceMinWorkerThreads=64 (verified present in the Worker's env) has no effect.

Behaves intermittently — some jobs complete; during bad spells nearly every job hangs at this point. Looks related to #2140 and the (closed) #1496.

Questions
  1. Is there a known deadlock in the results/live-log WebSocket setup path?
  2. Is there a client-side way to disable the live-log WebSocket feed (fall back to HTTP batch upload) as a workaround?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Runner.Worker log around ResultServer websocket creation and the JobServerQueue message, then inspect the Worker, ResultServer/WebSocket, and JobDispatcher paths mentioned in the report. Reproduce on a macOS/ARM64 self-hosted runner and compare a sample of the parked threads. Done means the worker proceeds past websocket setup, spawns the first job step, and no longer remains busy indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, macos
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.