actions / actions/runner

Self hosted runner jobs being cancelled after 24 hours when request to broker fails

Open
#3,869 5 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

A few weeks ago our actions that run for longer than 24 hours on our self hosted runners suddenly started being cancelled,

The operation was canceled.

After getting the runner and worker logs after rerunning with debug logging it looks like after roughly 24 hours in the runner log a request to the broker fails with a not found error,

[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound

Which prior to that all had been swimming along,

[2025-05-05 20:57:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:07:05 [2025-05-05 20:58:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:08:05 [2025-05-05 20:59:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:09:05 [2025-05-05 21:00:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:10:05 [2025-05-05 21:01:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:11:05 [2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound [2025-05-05 21:01:08Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished. [2025-05-05 21:01:08Z INFO JobDispatcher] Ensure WorkerDispatcher for job ca395085-040a-526b-2ce8-bdc85f692774 run to finish, cancel any running job. [2025-05-05 21:01:08Z INFO JobDispatcher] Send job cancellation message to worker for job ca395085-040a-526b-2ce8-bdc85f692774. [2025-05-05 21:01:08Z INFO ProcessChannel] Sending message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

Any help would be greatly appreciated!

To Reproduce

Run action for longer than 24 hours on self hosted runner

Expected behavior
Actions not cancelled abruptly after 24 hours

Runner Version and Platform

Latest 2.323 runner version and Ubuntu 24

What's not working?

[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing a self-hosted runner job lasting over 24 hours on Ubuntu and reviewing the runner logs around JobDispatcher and GitHubActionsService. Done means a long-running action is not abruptly cancelled when the broker request returns HTTP 404.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions, ubuntu
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.