Persistent runner drops the next job when it aborts its Busy long-poll at job completion ("was not acquired by Runner")

Describe the bug

When a job finishes, OnJobStatus (src/Runner.Listener/BrokerMessageListener.cs, and the same code in MessageListener.cs) sets _runnerStatus = Online and cancels _getMessagesTokenSource, aborting the GET /message?status=Busy long-poll that is still open. The service assigns the next queued job to the runner at that same moment and delivers it on that open poll. Aborting the request drops the message on the client, while the service treats it as delivered.

The service redelivers roughly 61 s later. By then the assignment has expired: acknowledge returns 404, acquirejob returns 409 MissingKey, and the runner logs "Skipping message Job". After repeated misses the job is cancelled with the check-run annotation:

The job was not acquired by Runner of type self-hosted even after multiple attempts

The job has no runner name and zero steps. Any aggregate job that needs: it then fails the workflow run.

Evidence (runner 2.337.0, Linux x64, persistent runner, repo-scoped)

[2026-09-19 01:41:43Z INFO Terminal] WRITE LINE: Job Release Validation (prod) completed with result: Succeeded
[2026-09-19 01:41:43Z WARN GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=...&status=Busy&runnerVersion=2.337.0... has been cancelled
[2026-09-19 01:41:43Z INFO BrokerMessageListener] Get messages has been cancelled using local token source. Continue to get messages with new status.
[2026-09-19 01:42:44Z INFO BrokerMessageListener] Acknowledging runner request '07f09524-...'.
[2026-09-19 01:42:44Z ERR  GitHubActionsService] POST request to https://broker.actions.githubusercontent.com/acknowledge?...&status=Online... failed. HTTP Status: NotFound
[2026-09-19 01:42:45Z ERR  GitHubActionsService] POST request to https://run-actions-3-azure-eastus.actions.githubusercontent.com/104/acquirejob failed. HTTP Status: Conflict
[2026-09-19 01:42:45Z INFO Runner] Skipping message Job. Job message already acquired '07f09524-...'. job assignment is invalid: MissingKey

A fresh status=Online poll opens at 01:41:43 but receives nothing until the 61 s redelivery. If the job had been queued service-side, that poll would have returned it immediately, so the first delivery went out on the aborted Busy poll.

Frequency

The client code is unchanged since 2.336.0 (our 2.336.0 logs show the same abort on every job completion). What changed is the rate, starting around 2026-09-13, which suggests a service-side change in when the next job is dispatched. MissingKey lines per day across two runners on one host:

Most misses are redelivered to the other runner in time. The rest end as cancelled, runner-less jobs: 6 in the last three days, all in a matrix that queues about a dozen jobs against two runners, which is where job-completion handoffs are densest.

To reproduce

Two persistent runners, one matrix of 10+ short jobs so the queue stays non-empty while jobs complete. Watch _diag/Runner_*.log for "cancelled using local token source" followed about 60 s later by acquirejob 409 MissingKey.

Expected behavior

A job assigned at completion time is received and run by the runner it was assigned to.

Suggested fix

Do not abort the in-flight poll on the Busy to Online transition. The service already knows the job completed, and it assigns work on the Busy poll, so letting that poll return delivers the job. _runnerStatus is still updated and goes out on the next poll. A draft PR follows.

Related

• #4617: the ephemeral-runner side of a lost acquire (409 MissingKey), where the runner turns into a ghost. This issue is the persistent-runner side, where the job is lost instead.

Runner version and platform

2.337.0 (behaviour present in 2.336.0), Linux x64, systemd-managed persistent runners, repository scope.

Start with OnJobStatus in src/Runner.Listener/BrokerMessageListener.cs and the corresponding code in MessageListener.cs, then reproduce with two persistent runners and a queue of short matrix jobs while watching diag/Runner*.log. Verify that a job assigned during completion is received and run by the runner rather than being lost after the Busy long-poll is cancelled.