actions / actions/runner

Self hosted runner are going offline suddenly and it is not connected.

Open
#3,501 18 comments 15 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
We are using self hosted macos machines for our Github actions. It was working fine till yesterday but its suddenly getting disconnected and going to offline. And when I check the runner logs in _diag, I did find this

What's not working?

[2024-10-10 21:30:29Z INFO Terminal] WRITE LINE:
[2024-10-10 21:30:29Z INFO RSAFileKeyManager] Loading RSA key parameters from file /System/Volumes/Data/export/home/tester/actions-runner/.credentials_rsaparams
[2024-10-10 21:30:30Z ERR  GitHubActionsService] POST request to https://tokenghub.actions.githubusercontent.com/_apis/oauth2/token/****** failed. HTTP Status: BadRequest
[2024-10-10 21:30:30Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2024-10-10 21:30:30Z ERR  MessageListener] Catch exception during create session.
[2024-10-10 21:30:30Z ERR  MessageListener] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration was not found or is not medium trust. ClientType:
   at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
   at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at GitHub.Runner.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
[2024-10-10 21:30:30Z ERR  MessageListener] Test oauth app registration.
[2024-10-10 21:30:30Z INFO RSAFileKeyManager] Loading RSA key parameters from file /System/Volumes/Data/export/home/tester/actions-runner/.credentials_rsaparams
[2024-10-10 21:30:30Z ERR  GitHubActionsService] POST request to https://tokenghub.actions.githubusercontent.com/_apis/oauth2/token/***** failed. HTTP Status: BadRequest
[2024-10-10 21:30:30Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[2024-10-10 21:30:30Z INFO MessageListener] Retriable exception: Registration was not found or is not medium trust. ClientType:
[2024-10-10 21:30:30Z INFO MessageListener] Sleeping for 30 seconds before retrying.

I have been seeing this issue from couple of days and its happening for all our hosts.

I did see some online thread, some cases were clock of the server was off( which is not true in our case)

Runner Version and Platform

Runner version we are using is the latest 2.319.1
OS of the machine running the runner? OSX

Runner and Worker's Diagnostic Logs

Added the runner logs above.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the _diag output and the MessageListener.CreateSessionAsync stack entry, focusing on the OAuth token request returning BadRequest and “Registration was not found or is not medium trust.” Reproduce or trace the failure on the reported macOS runner version 2.319.1, then verify that affected self-hosted runners remain connected without repeating the error.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions, macos
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.