cache: concurrent download is gated on the .blob.core.windows.net hostname, so every non-Azure cache backend restores single-stream
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 5.9k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
Summary
@actions/cache selects its download strategy from the archive URL's hostname (packages/cache/src/internal/cacheHttpClient.ts, downloadCache):
if (archiveUrl.hostname.endsWith('.blob.core.windows.net')) {
// concurrent ranged download (downloadCacheHttpClientConcurrent / storage SDK)
} else {
await downloadCacheHttpClient(archiveLocation, archivePath) // one stream, no Range, no keepAlive
}
Any cache backend that is not Azure Blob — every self-hosted implementation of the results API, and GHES deployments fronting their own storage — is pinned to one HTTP stream. downloadConcurrency / concurrentBlobDownloads are computed and discarded; restoreCacheV2 sets useAzureSdk: true but that flag is only read inside the Azure branch. There is no input or environment variable that changes this.
Measurements
Self-hosted runner (Kubernetes pod, Scaleway), one 320 MB cache object served from S3-compatible object storage via a presigned URL:
| path | throughput |
|---|---|
actions/cache restore (today) |
~15 MB/s |
curl single stream, same URL |
54 MB/s |
8 parallel 40 MB Range GETs, same URL |
143 MB/s |
We since built a drop-in action that reuses @actions/cache's own twirp client, getCacheVersion and extractTar and swaps only the downloader for a parallel ranged one. Same 320 MB entry, cold runner per leg, sha256-verified: actions/cache/restore 5.6 s; ranged 3.5 s with the transfer itself at 140 MB/s over 9 requests. The object store, network and URL are fine; the client shape is the limit. Blacksmith has documented the same gate and worked around it by rewriting hostnames to Azure-shaped ones (their write-up on cache speed).
Why #1943 does not fix it
actions/toolkit#1943 lifts useAzureSdk out of the check but keeps archiveUrl.hostname.endsWith('.blob.core.windows.net') as a condition for the concurrent HTTP path; the only new path it opens for non-Azure hosts is downloadCacheStorageSDK, which speaks the Azure Blob protocol (BlockBlobClient) and cannot be pointed at an S3 endpoint.
Suggested fix
Make the concurrent HTTP downloader (downloadCacheHttpClientConcurrent) usable regardless of hostname — for example when concurrentBlobDownloads is set, or via a DownloadOptions flag / env var — and let the server's Accept-Ranges / a Range probe decide, rather than the hostname. Two details for non-Azure backends:
- presigned S3 URLs are typically signed for
GETonly, so the initialHEADused forcontent-lengthanswers 403; aRange: bytes=0-0GET returns the total incontent-rangewithout needing HEAD; - 4 MiB blocks are far below the per-request overhead of object stores: 8×4 MiB measured 17 MB/s against 143 MB/s for 8×40 MB on the same object. A configurable block size, or sizing by object length, matters.
Happy to open a PR along these lines if maintainers agree on the shape.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/cache/src/internal/cacheHttpClient.ts at downloadCache and trace downloadCacheHttpClientConcurrent, downloadCacheHttpClient, and the concurrentBlobDownloads/useAzureSdk options. Compare the hostname gate with the suggested Range-based behavior, including presigned URLs that reject HEAD and configurable block sizing. Done means non-Azure cache URLs can use concurrent ranged downloads without breaking the existing Azure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100