actions / actions/runner

Runner's Azure Blob uploads stall through HTTPS proxy — BlobClient not configured with proxy transport

Open
#4,351 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Description

When the GitHub Actions runner is configured with HTTPS_PROXY, uploads to Azure Blob Storage (step logs, summaries, artifacts via the Results Service) stall for ~75 seconds and timeout. The runner's ResultsHttpClient creates BlobClient / AppendBlobClient instances without configuring an HttpClientTransport that uses the proxy, so the Azure SDK falls back to its default transport which may not inherit the runner's proxy configuration correctly.

A standalone .NET 8.0 HttpClient.PutAsync() through the same proxy to the same blob endpoints completes in ~1 second — the issue is specific to how the runner creates Azure SDK blob clients.

Root cause analysis

In src/Sdk/WebApi/WebApi/ResultsHttpClient.cs, the GetBlobClient() and GetAppendBlobClient() methods create blob clients with only retry/timeout options:

private BlobClient GetBlobClient(string url)
{
    var blobUri = ParseSasToken(url);
    var opts = new BlobClientOptions
    {
        Retry =
        {
            MaxRetries = Constants.DefaultBlobUploadRetries,
            NetworkTimeout = TimeSpan.FromSeconds(Constants.DefaultNetworkTimeoutInSeconds)
        }
    };
    return new BlobClient(blobUri.path, new AzureSasCredential(blobUri.sas), opts);
}

No HttpClientTransport is provided in BlobClientOptions, so the Azure SDK uses its default HTTP pipeline. Meanwhile, the runner's own HTTP calls use a custom RunnerWebProxy via VssHttpMessageHandler, but this proxy configuration is not shared with the Azure SDK blob clients.

Suggested fix

Configure the BlobClientOptions.Transport with an HttpClientTransport that uses the runner's proxy settings:

private BlobClient GetBlobClient(string url)
{
    var blobUri = ParseSasToken(url);
    var handler = new HttpClientHandler();
    // Use the runner's proxy, or let the default pick up HTTPS_PROXY env var
    handler.Proxy = WebProxy;  // RunnerWebProxy instance
    handler.UseProxy = true;

    var opts = new BlobClientOptions
    {
        Retry =
        {
            MaxRetries = Constants.DefaultBlobUploadRetries,
            NetworkTimeout = TimeSpan.FromSeconds(Constants.DefaultNetworkTimeoutInSeconds)
        },
        Transport = new HttpClientTransport(handler)
    };
    return new BlobClient(blobUri.path, new AzureSasCredential(blobUri.sas), opts);
}

The same fix is needed for GetAppendBlobClient().

Evidence

Standalone .NET repro — works fine (no stall):

$ dotnet run
HTTPS_PROXY = http://10.129.160.8:443
.NET        = 8.0.26

--- PUT https://productionresultssa6.blob.core.windows.net/test-upload ---
HTTP 409 in 1.1s

--- PUT https://results-receiver.actions.githubusercontent.com/test ---
HTTP 404 in 0.2s

Same proxy, same endpoints — curl and Python also work:

TEST 1a: curl PUT 1MB to blob → HTTP 409 in 0.3s
TEST 2:  Python PUT 1MB to blob → HTTP 409 in 0.6s

The runner process stalls on the same endpoints:
From GCP Secure Web Proxy logs during a real workflow run:

host=productionresultssa6.blob.core.windows.net:443  latency=74.999634s  ua=azsdk-net-Storage.Blobs/12.27.0 (.NET 8.0.25)
host=results-receiver.actions.githubusercontent.com:443  latency=74.999750s  ua=VSServices/2.333.1
host=results-receiver.actions.githubusercontent.com:443  latency=74.998897s  ua=VSServices/2.333.1
host=results-receiver.actions.githubusercontent.com:443  latency=74.999444s  ua=VSServices/2.333.1

Each request stalls for exactly 75 seconds (the proxy's tunnel timeout), indicating the runner is establishing the CONNECT tunnel but then failing to send/receive data through it.

Affected operations

The stall affects these runner-internal operations that use BlobClient / AppendBlobClient:

  • UploadStepSummaryAsync — step summary upload
  • UploadResultsStepLogAsync — step log upload
  • UploadResultsJobLogAsync — job log upload
  • UploadResultsDiagnosticLogsAsync — diagnostic log upload

The upload-artifact action itself is Node.js and uses the JavaScript Azure Storage SDK — it is not affected. The stall comes from the runner process's own results/log uploads.

Environment

  • Runner: GitHubActionsRunner-linux-x64/2.333.1
  • Runtime: .NET 8.0.25 (self-contained)
  • Azure SDK: azsdk-net-Storage.Blobs/12.27.0
  • Proxy: GCP Secure Web Proxy (HTTPS forward proxy, CONNECT tunnel, no TLS inspection)
  • Config: HTTPS_PROXY=http://<proxy-ip>:443
  • Platform: Ubuntu 24.04.4 LTS on GKE (actions-runner-controller v0.13.1)

Workaround

Add the affected hostnames to NO_PROXY to bypass the proxy:

NO_PROXY=.blob.core.windows.net,.actions.githubusercontent.com

This is not ideal because it requires direct internet access from the runner nodes, bypassing the proxy's hostname-based egress control.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read src/Sdk/WebApi/WebApi/ResultsHttpClient.cs, focusing on GetBlobClient() and GetAppendBlobClient(), then inspect RunnerWebProxy.cs to understand the runner's proxy configuration. Reproduce the affected upload requests through the HTTPS proxy; done means the BlobClient and AppendBlobClient operations complete without the recurring 75-second stall.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.