Self hosted runner are going offline suddenly and it is not connected.
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
We are using self hosted macos machines for our Github actions. It was working fine till yesterday but its suddenly getting disconnected and going to offline. And when I check the runner logs in _diag, I did find this
What's not working?
[2024-10-10 21:30:29Z INFO Terminal] WRITE LINE:
[2024-10-10 21:30:29Z INFO RSAFileKeyManager] Loading RSA key parameters from file /System/Volumes/Data/export/home/tester/actions-runner/.credentials_rsaparams
[2024-10-10 21:30:30Z ERR GitHubActionsService] POST request to https://tokenghub.actions.githubusercontent.com/_apis/oauth2/token/****** failed. HTTP Status: BadRequest
[2024-10-10 21:30:30Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2024-10-10 21:30:30Z ERR MessageListener] Catch exception during create session.
[2024-10-10 21:30:30Z ERR MessageListener] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration was not found or is not medium trust. ClientType:
at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
at GitHub.Runner.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
[2024-10-10 21:30:30Z ERR MessageListener] Test oauth app registration.
[2024-10-10 21:30:30Z INFO RSAFileKeyManager] Loading RSA key parameters from file /System/Volumes/Data/export/home/tester/actions-runner/.credentials_rsaparams
[2024-10-10 21:30:30Z ERR GitHubActionsService] POST request to https://tokenghub.actions.githubusercontent.com/_apis/oauth2/token/***** failed. HTTP Status: BadRequest
[2024-10-10 21:30:30Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2024-10-10 21:30:30Z INFO MessageListener] Retriable exception: Registration was not found or is not medium trust. ClientType:
[2024-10-10 21:30:30Z INFO MessageListener] Sleeping for 30 seconds before retrying.
I have been seeing this issue from couple of days and its happening for all our hosts.
I did see some online thread, some cases were clock of the server was off( which is not true in our case)
Runner Version and Platform
Runner version we are using is the latest 2.319.1
OS of the machine running the runner? OSX
Runner and Worker's Diagnostic Logs
Added the runner logs above.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the _diag output and the MessageListener.CreateSessionAsync stack entry, focusing on the OAuth token request returning BadRequest and “Registration was not found or is not medium trust.” Reproduce or trace the failure on the reported macOS runner version 2.319.1, then verify that affected self-hosted runners remain connected without repeating the error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions, macos
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100