Ephemeral runner hangs forever after "Acknowledging runner request": acquirejob never returns, no Runner.Worker, no error, runner never exits

Describe the bug

An ephemeral runner (v2.337.0, ARC gha-runner-scale-set 0.14.2) receives a job, logs Acknowledging runner request '<id>', reloads credentials, logs one AAD Correlation ID for this token request: Unknown line, and then does nothing, ever:

• No Runner.Worker is spawned and no Worker_*.log is written.
• No error or retry is logged. RunServer never logs Catch exception during request or Back off ..., so no exception was raised: the await on acquirejob simply never completes.
• Runner.Listener stays alive (Sl, 0 % CPU) holding one ESTABLISHED, idle HTTPS connection. It does not resume long-polling the broker.
• Exactly 10 minutes later GitHub fails the job with "The self-hosted runner lost communication with the server...". No step ever started.
• The pod stays Running indefinitely (5.5 h for the oldest when we found it). ARC still counts it as a live replica, so each hung runner permanently removes one slot of capacity (see ARC impact below).

We hit this four times on 2026-09-02, in two pairs a few minutes apart within each pair. Dozens of other jobs on the same scale set ran normally before, between and after.

This is distinct from the existing "ghost ephemeral runner" reports, all of which get an HTTP error back from acquirejob and log it:

• #4617 / PR #4618: acquirejob returns 409, runner logs Skipping message Job and continues.
• actions/actions-runner-controller#4307: acquirejob returns 404 after a cancellation (job was canceled), same continue.
• #4569 / #4312: hang in the cancellation redelivery loop, with a running worker.
• #4598: worker is spawned but never reads the IPC message; listener exits 0.

Here nothing comes back at all. Reading the client code (below), the request has no deadline on the response body and no TCP keepalive, so a connection that dies after headers but before the body completes can block forever. The same day, another ARC 0.14.2 user reported 25 runners wedged with the same last log line during ~40 min of packet loss to the Actions broker (actions/actions-runner-controller#4396, comment of 2026-09-02), which is consistent with a network event on the path being the trigger.

To Reproduce

Not deterministic. Observed setup:

1. Org-level ARC scale set (minRunners: 3, maxRunners: 14), ephemeral runners, dind sidecar, GitHub App auth.
2. Normal job load. Two runners that acknowledge a request within a few minutes of each other hang at the same point; ~80 minutes later, two more.
3. GitHub fails the four jobs after 10 minutes; the four pods stay Running for hours.

Expected behavior

• acquirejob should complete or fail within a bounded time (Settings.SendTimeout, default 100 s), covering the response body as well as the headers.
• An ephemeral runner that has acknowledged a request and then cannot acquire it has no future. It should exit so the orchestrator can replace it, the way #4540 already made it exit when the acknowledge itself returns job-not-found.
• A Runner.Listener sitting for hours with no worker, no error and no broker polling is a hang, whatever the cause.

Runner Version and Platform

• Runner: 2.337.0 (ghcr.io/actions/actions-runner:latest at the time), Linux x64 container. The relevant code is unchanged on main, so this is not specific to 2.337.0.
• ARC: gha-runner-scale-set 0.14.2, controller 0.14.2
• Kubernetes: k3s v1.33.13+k3s2, containerd 2.2.5, single bare-metal node (96 threads / 251 GiB), kernel 7.0.0
• Pod: dind via native sidecar (docker:dind as initContainer with restartPolicy: Always), runner container privileged: true, ACTIONS_RUNNER_HOOK_JOB_STARTED set, RUNNER_WAIT_FOR_DOCKER_IN_SECONDS=120. The dind sidecar was healthy in every affected pod.
• Auth: GitHub App, org-level (githubConfigUrl: https://github.com/<org>), runner group Default.

What's not working?

Occurrences (all 2026-09-02, UTC):

runner pod	acknowledged	GitHub failed job	pod still Running at
runner-c8bqp	14:11:37	14:21:49	19:40+
runner-x5swc	~14:15:37	~14:25	19:40+
runner-sjdqk	15:33:07	15:43:12	19:40+
runner-j69ht	15:39:58	15:50:03	19:40+

GitHub's annotation on all four jobs:

The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

ARC recorded JobStarted for each runner (the EphemeralRunner status has jobId, workflowRunId and jobRepositoryName filled in), so the broker did consider the job dispatched.

State inside runner-c8bqp about 5.5 h after the acknowledge:

$ ps -eo pid,etime,stat,args
    PID     ELAPSED STAT COMMAND
      1    05:44:12 Ss   /bin/bash /home/runner/run.sh
     10    05:44:12 S    /bin/bash /home/runner/run-helper.sh
     45    05:44:12 Sl   /home/runner/bin/Runner.Listener run

$ ls /home/runner/_diag
Runner_20260902-135814-utc.log          # last write 14:11; no Worker_*.log exists

$ cat /proc/net/tcp6   # one ESTABLISHED connection to an Azure IP :443, nothing else

Impact on ARC

ARC's listener computes target = min(minRunners + TotalAssignedJobs, maxRunners). GitHub correctly stops counting the failed job, but the EphemeralRunnerSet still counts the hung pod as a live replica. With N hung pods, any job that arrives while assigned <= N never gets a runner because the target is "already met". With four hung pods and minRunners: 3 we had one job queued 46 min and another 60 min on a host that was 90 % idle. Both were only served when an unrelated burst pushed the target above the zombie count. Listener log during the second incident, repeating every ~50 s for an hour:

"Calculated target runner count" "assigned job"=1 decision=4 min=3 max=14 currentRunnerCount=4

Job Log Output

Empty. No step of any of the four jobs started; the only content is the annotation quoted above.

Runner and Worker's Diagnostic Logs

Complete tail of the runner container stdout for runner-c8bqp. Nothing follows for 5.5 h. The _diag/Runner_*.log ends at the same line. There is no Worker_*.log.

[RUNNER 2026-09-02 13:58:18Z INFO Terminal] WRITE LINE: Current runner version: '2.337.0'
[RUNNER 2026-09-02 13:58:18Z INFO Terminal] WRITE LINE: 2026-09-02 13:58:18Z: Listening for Jobs
[RUNNER 2026-09-02 13:58:18Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2026-09-02 14:11:37Z INFO BrokerMessageListener] Acknowledging runner request '1163817b-8f76-50d5-a595-c971db5244ea'.
[RUNNER 2026-09-02 14:11:41Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2026-09-02 14:11:41Z INFO ConfigurationStore] stored True
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] Creating type OAuth
[RUNNER 2026-09-02 14:11:41Z INFO CredentialManager] Creating credential type: OAuth
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2026-09-02 14:11:41Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown

The other three pods show the same lines with their own request IDs.

We still have live hung pods and can provide a dotnet-dump / createdump of Runner.Listener, the full _diag logs, or run a debug build, on request.

Analysis (actions/runner at tag v2.337.0, commit 397b032; same on main)

The broker message loop handles a run-service job request inline, so a stuck acquire freezes the whole listener: no further broker polling, no exit.

src/Runner.Listener/Runner.cs:

1. L697 AcknowledgeMessageAsync (5 s timeout) logs Acknowledging runner request. Seen.
2. L699-704: an ephemeral runner exits if the acknowledge returns job-not-found (#4540). Not our path.
3. L731-733 LoadCredentials(allowAuthUrlV2: true) + runServer.ConnectAsync(...). No HTTP.
4. L735 runServer.GetJobMessageAsync(...) issues POST .../acquirejob. This never returned.
5. L738-746 catch 404/409/422 and continue; L747-759 catch anything else, log Caught exception from acquiring job message and continue. Neither ran: nothing was logged.
6. L762 jobDispatcher.Run(...) never reached, hence no Runner.Worker.

GetJobMessageAsync (src/Runner.Common/RunServer.cs:63-72) wraps the call in RetryRequest (src/Runner.Common/RunnerService.cs:85-108), which logs Catch exception during request and Back off ... on any failure. Nothing was logged, so no exception was raised: the await never completed.

Why the await can never complete: the request timeout covers the headers but not the body.

• src/Sdk/WebApi/WebApi/RawHttpClientBase.cs:66: the constructor sets m_client.Timeout = TimeSpan.FromMilliseconds(-1.0) with the comment "Disable their timeout since we handle it ourselves".
• RawHttpClientBase.cs:214-217, 245: requests are sent with HttpCompletionOption.ResponseContentRead, so HttpClient.SendAsync buffers the whole body before returning, bounded only by that infinite timeout and the caller's cancellation token.
• The timeout "handled ourselves" lives in src/Sdk/Common/Common/RawHttpMessageHandler.cs: L141-143 tokenSource.CancelAfter(Settings.SendTimeout) (100 s by default, src/Runner.Sdk/Util/VssUtil.cs:79-86) applies to m_tokenProvider.GetTokenAsync (L159, which emitted the final AAD Correlation ID log line) and to m_messageInvoker.SendAsync(request, tokenSource.Token) (L171). A message handler returns as soon as the response headers arrive; the finally at L231 then disposes the token source and its timer. The body is read afterwards by HttpClient, outside that timer.
• The caller's token, messageQueueLoopTokenSource.Token, is only cancelled on shutdown or session restart. So the body read has no deadline at all.
• No TCP keepalive is configured on this client. The only keepalive code in the tree (src/Sdk/WebApi/WebApi/VssHttpClientBase.cs:1153-1163) goes through ServicePointManager.SetTcpKeepAlive, which is a no-op for SocketsHttpHandler on .NET 8 (the runner targets net8.0), and RawHttpClientBase, the client used for acquirejob, does not call it anyway.

Put together: if the connection carrying the acquirejob response dies after the status line and headers but before the body (the full job message, often tens to hundreds of KB) is complete, and the peer's RST/FIN never reaches the runner (half-open connection: middlebox or LB state loss, NAT table churn, path outage), the read blocks forever. No timer exists to cancel it and no keepalive probe exists to detect the dead peer, so the ephemeral runner sits with one ESTABLISHED idle socket and never exits. That is exactly the process state we observed. Two runners hanging within minutes of each other, twice in one day, fits a transient network event rather than a per-job trigger. We have not yet confirmed the blocked frame with a dump; happy to do so.

VssHttpClientBase has the same infinite m_client.Timeout at L76, so the broker long-poll client has the same exposure.

Workaround we deployed

A CronJob deletes any EphemeralRunner in phase Running with status.jobId set, pod age > 12 min, and no Runner.Worker process in the runner container on two consecutive checks. Deleting the EphemeralRunner makes ARC deregister the runner and backfill the slot. Others in actions/actions-runner-controller#4307 use activeDeadlineSeconds on the pod or a log-grepping liveness probe; neither catches this variant, since nothing is logged.

Start with GetJobMessageAsync in src/Runner.Common/RunServer.cs and follow RetryRequest in src/Runner.Common/RunnerService.cs into RawHttpClientBase.cs and RawHttpMessageHandler.cs. Trace how response headers, body reads, cancellation, and Settings.SendTimeout interact. Done means acquirejob cannot hang indefinitely and an ephemeral runner exits when it cannot acquire an acknowledged job.