Runner connect error after successful Github authentication

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

v0.25.2

Deployment Method

Other - kubectl apply

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Create a private EKS cluster running Kubernetes v1.27 and using IPv4 cluster IP address family. I created the cluster with a managed node group of 3 nodes of `t3.medium`.
2. Setup an AWS Client VPN to be able to communicate with the cluster API inside the private subnet. 
3. Create a Github App for my organization with appropriate permissions (as detailed in the docs).
3. Connect to the Kubernetes API via VPN.
4. Install `cert-manager` on the cluster.
5. Create the K8s secret `controller-manager` with the relevant fields for the Github app including app_id, app_installation_id, app_private_key.
6. Deploy ARC using `kubectl create -f https://github.com/actions/actions-runner-controller/releases/download/v0.25.2/actions-runner-controller.yaml`
7. Create a RunnerDeployment with the following:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  replicas: 1
  template:
    spec:
      repository: chariot-giving/chariot

8. Tail the pod logs. I see the pod successfully authenticates to Github and I even can confirm the runner is registered on Github (I can see it `Idle` state). After 1 minute, the status changes to `Offline` and I see the following in the logs:

Current runner version: '2.310.2'
2023-10-19 18:45:03Z: Listening for Jobs
2023-10-19 18:46:54Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

Describe the bug

The problem is that the runner disconnects from Github after 1 minute after it had previously authenticated and connected to Github.

• I've checked the network connection between the ec2 instance and Github's servers via AWS Reachability Analyzer and it confirms what I see initially - that the network is reachable.
• Every time I delete the pod and new one spins up, it reconnects to Github successfully and then eventually times out after 1 minute.
• I've been able to schedule a Github actions workflow job to the runner (immediately after I restarted it and it was still connected to Github) however the job failed to finish after the runner disconnected -> offline.
• I even exec'd into the runner pod and executed nslookup github.com and curl -v https://github.com and what I found was that before the connect time out, both commands returned successfully, however AFTER the connect time out, both commands fail.

Describe the expected behavior

The Github Runner stays connected to Github.

Additional Context

Here is the runner diagnostic logs:

[2023-10-19 19:24:27Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:24:47Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.802 seconds.
[2023-10-19 19:25:18Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.757 seconds.
[2023-10-19 19:25:51Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 17.742 seconds.
[2023-10-19 19:26:29Z ERR  GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The maximum number of attempts has been reached.
[2023-10-19 19:26:29Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2023-10-19 19:26:29Z INFO RunnerServer] Catch exception during connect. 4 attempt left.
[2023-10-19 19:26:29Z ERR  RunnerServer] System.Net.Http.HttpRequestException: Resource temporarily unavailable (pipelinesghubeus8.actions.githubusercontent.com:443)
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2023-10-19 19:26:29Z ERR  RunnerServer] #####################################################
[2023-10-19 19:26:29Z ERR  RunnerServer] System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2023-10-19 19:26:29Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:26:49Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.825 seconds.
[2023-10-19 19:27:20Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.805 seconds.
[2023-10-19 19:27:52Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 15.922 seconds.
[2023-10-19 19:28:28Z ERR  GitHubActionsService] Attempt 4 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The maximum number of attempts has been reached.
[2023-10-19 19:28:28Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2023-10-19 19:28:28Z INFO RunnerServer] Catch exception during connect. 3 attempt left.
[2023-10-19 19:28:28Z ERR  RunnerServer] System.Net.Http.HttpRequestException: Resource temporarily unavailable (pipelinesghubeus8.actions.githubusercontent.com:443)
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at GitHub.Runner.Common.RunnerService.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
[2023-10-19 19:28:28Z ERR  RunnerServer] #####################################################
[2023-10-19 19:28:28Z ERR  RunnerServer] System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
[2023-10-19 19:28:29Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2023-10-19 19:28:49Z WARN GitHubActionsService] Attempt 1 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 10.964 seconds.
[2023-10-19 19:29:20Z WARN GitHubActionsService] Attempt 2 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 12.91 seconds.
[2023-10-19 19:29:52Z WARN GitHubActionsService] Attempt 3 of GET request to https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1 failed (Socket Error: TryAgain). The operation will be retried in 15.782 seconds.

### Controller Logs

```shell
https://gist.github.com/magaldima/8af6b4db0e724ed5c87e9ba7dc558f0c

Runner Pod Logs

Defaulted container "runner" out of: runner, docker
2023-10-19 18:44:56.996  NOTICE --- Runner init started with pid 8
2023-10-19 18:44:57.8  DEBUG --- Github endpoint URL https://github.com/
2023-10-19 18:44:57.878  DEBUG --- Passing --ephemeral to config.sh to enable the ephemeral runner.
2023-10-19 18:44:57.889  DEBUG --- Configuring the runner.

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration




√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.

2023-10-19 18:45:01.895  DEBUG --- Runner successfully configured.
{
  "agentId": 30,
  "agentName": "example-runnerdeploy-xzx8m-99d4f",
  "poolId": 1,
  "poolName": "Default",
  "ephemeral": true,
  "serverUrl": "https://pipelinesghubeus8.actions.githubusercontent.com/bsi4JQ9FlN18axuGZNtTAkAXAV1aIZOYIFDLhCbMcsaiQLxRrR/",
  "gitHubUrl": "https://github.com/chariot-giving/chariot",
  "workFolder": "/runner/_work"
2023-10-19 18:45:01.901  DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2023-10-19 18:45:01.903  DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
}CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
2023-10-19 18:45:01.998  NOTICE --- WARNING LATEST TAG HAS BEEN DEPRECATED. SEE GITHUB ISSUE FOR DETAILS:
2023-10-19 18:45:02.5  NOTICE --- https://github.com/actions/actions-runner-controller/issues/2056

√ Connected to GitHub

Current runner version: '2.310.2'
2023-10-19 18:45:03Z: Listening for Jobs
2023-10-19 18:46:54Z: Runner connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

Start with the reproduction steps and the runner diagnostic logs, focusing on the transition from successful authentication to the connection timeout against pipelinesghubeus8.actions.githubusercontent.com. Done means the runner remains connected and online after the initial authentication instead of becoming Offline after one minute.