Self hosted runner jobs being cancelled after 24 hours when request to broker fails
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
A few weeks ago our actions that run for longer than 24 hours on our self hosted runners suddenly started being cancelled,
The operation was canceled.
After getting the runner and worker logs after rerunning with debug logging it looks like after roughly 24 hours in the runner log a request to the broker fails with a not found error,
[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound
Which prior to that all had been swimming along,
[2025-05-05 20:57:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:07:05 [2025-05-05 20:58:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:08:05 [2025-05-05 20:59:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:09:05 [2025-05-05 21:00:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:10:05 [2025-05-05 21:01:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:11:05 [2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound [2025-05-05 21:01:08Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished. [2025-05-05 21:01:08Z INFO JobDispatcher] Ensure WorkerDispatcher for job ca395085-040a-526b-2ce8-bdc85f692774 run to finish, cancel any running job. [2025-05-05 21:01:08Z INFO JobDispatcher] Send job cancellation message to worker for job ca395085-040a-526b-2ce8-bdc85f692774. [2025-05-05 21:01:08Z INFO ProcessChannel] Sending message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
Any help would be greatly appreciated!
To Reproduce
Run action for longer than 24 hours on self hosted runner
Expected behavior
Actions not cancelled abruptly after 24 hours
Runner Version and Platform
Latest 2.323 runner version and Ubuntu 24
What's not working?
[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing a self-hosted runner job lasting over 24 hours on Ubuntu and reviewing the runner logs around JobDispatcher and GitHubActionsService. Done means a long-running action is not abruptly cancelled when the broker request returns HTTP 404.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions, ubuntu
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100