actions / actions/toolkit

cache: concurrent download is gated on the .blob.core.windows.net hostname, so every non-Azure cache backend restores single-stream

Open
#2,487 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
5.9k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

Summary

@actions/cache selects its download strategy from the archive URL's hostname (packages/cache/src/internal/cacheHttpClient.ts, downloadCache):

if (archiveUrl.hostname.endsWith('.blob.core.windows.net')) {
  // concurrent ranged download (downloadCacheHttpClientConcurrent / storage SDK)
} else {
  await downloadCacheHttpClient(archiveLocation, archivePath) // one stream, no Range, no keepAlive
}

Any cache backend that is not Azure Blob — every self-hosted implementation of the results API, and GHES deployments fronting their own storage — is pinned to one HTTP stream. downloadConcurrency / concurrentBlobDownloads are computed and discarded; restoreCacheV2 sets useAzureSdk: true but that flag is only read inside the Azure branch. There is no input or environment variable that changes this.

Measurements

Self-hosted runner (Kubernetes pod, Scaleway), one 320 MB cache object served from S3-compatible object storage via a presigned URL:

path throughput
actions/cache restore (today) ~15 MB/s
curl single stream, same URL 54 MB/s
8 parallel 40 MB Range GETs, same URL 143 MB/s

We since built a drop-in action that reuses @actions/cache's own twirp client, getCacheVersion and extractTar and swaps only the downloader for a parallel ranged one. Same 320 MB entry, cold runner per leg, sha256-verified: actions/cache/restore 5.6 s; ranged 3.5 s with the transfer itself at 140 MB/s over 9 requests. The object store, network and URL are fine; the client shape is the limit. Blacksmith has documented the same gate and worked around it by rewriting hostnames to Azure-shaped ones (their write-up on cache speed).

Why #1943 does not fix it

actions/toolkit#1943 lifts useAzureSdk out of the check but keeps archiveUrl.hostname.endsWith('.blob.core.windows.net') as a condition for the concurrent HTTP path; the only new path it opens for non-Azure hosts is downloadCacheStorageSDK, which speaks the Azure Blob protocol (BlockBlobClient) and cannot be pointed at an S3 endpoint.

Suggested fix

Make the concurrent HTTP downloader (downloadCacheHttpClientConcurrent) usable regardless of hostname — for example when concurrentBlobDownloads is set, or via a DownloadOptions flag / env var — and let the server's Accept-Ranges / a Range probe decide, rather than the hostname. Two details for non-Azure backends:

  • presigned S3 URLs are typically signed for GET only, so the initial HEAD used for content-length answers 403; a Range: bytes=0-0 GET returns the total in content-range without needing HEAD;
  • 4 MiB blocks are far below the per-request overhead of object stores: 8×4 MiB measured 17 MB/s against 143 MB/s for 8×40 MB on the same object. A configurable block size, or sizing by object length, matters.

Happy to open a PR along these lines if maintainers agree on the shape.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/cache/src/internal/cacheHttpClient.ts at downloadCache and trace downloadCacheHttpClientConcurrent, downloadCacheHttpClient, and the concurrentBlobDownloads/useAzureSdk options. Compare the hostname gate with the suggested Range-based behavior, including presigned URLs that reject HEAD and configurable block sizing. Done means non-Azure cache URLs can use concurrent ranged downloads without breaking the existing Azure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.