actions / actions/runner

Action archive downloads buffer the entire tarball in memory before writing to disk

Open Beginner friendly
#4,587 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
Runner.Worker downloads action repository archives with HttpClient.GetAsync(downloadUrl) using the default HttpCompletionOption.ResponseContentRead, which buffers the entire response body in the managed heap before ReadAsStreamAsync() returns (see code)

The subsequent CopyToAsync into the FileStream therefore copies from a memory buffer, not from the network, so the streaming-to-disk structure of this code never actually streams.

For actions hosted in large repositories, the codeload tarball is a snapshot of the repository's entire source tree at the referenced commit, which adds up to ~73 MB compressed in our case for a monorepo hosting reusable workflows/composite actions, so peak worker memory grows by the full archive size per download. We found out the hard way when we tried to set a low limit on the .NET heap in order to optimize our CI's memory utilization and pack more runners per k8s node. On failure, the retry loop re-buffers the entire body up to 3 times back-to-back (see the "Job Log Output" section below).

Even where it doesn't OOM, this forces every runner in the world to provision memory headroom of baseline + largest action archive per concurrent job, and puts multi-MB allocations on the Large Object Heap on every uncached action download. A typical job references 2–5 actions. Most marketplace actions are small, for example actions/checkout is a ~1MB tarball, so the buffering waste there is a few MB per download, mostly invisible. But worldwide it adds up quickly, and GitHub Actions have been gaining in popularity and over 2B workflow runs/month, so even small changes add up. Say conservatively 3 uncached downloads per run at ~2MB median: that's ~12 petabytes/month of needless transient heap allocation across the fleet. Given that RAM is now a luxury, it would make sense to be more nimble here.

Related reports of runner memory footprint pressure: #747 (request to support 512MB machines), #3796 (listener memory usage) — this is one of the concrete reasons the worker's peak memory is much larger than expected.

To Reproduce
Steps to reproduce the behavior:

  1. Run a self-hosted runner in a container with a constrained heap, e.g. DOTNET_GCHeapHardLimit=0x4000000 (64MiB) or a ~256Mi cgroup memory limit.
  2. Run a workflow with a uses: step referencing an action that lives in a large repository (tarball >> heap budget).
  3. The "Download action repository" phase fails with System.OutOfMemoryException (or, without a heap cap, worker RSS spikes by the archive size).

Expected behavior
The archive is streamed from the network to disk with a fixed-size buffer, keeping peak memory independent of archive size.

Runner Version and Platform

Version of your runner?

2.336.0 (also present at HEAD, 34ef7f2) on Linux, containerized self-hosted runner (ghcr.io/actions/actions-runner base) on Kubernetes.

What's not working?

.NET runtime OOMs during the download phase with small heaps.

Job Log Output

Warning: Failed to download action 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>'. Error: Exception of type 'System.OutOfMemoryException' was thrown.
Warning: Back off 17.072 seconds before retry.
Warning: Failed to download action 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>'. Error: Exception of type 'System.OutOfMemoryException' was thrown.
Warning: Back off 20.037 seconds before retry.
Error: Exception of type 'System.OutOfMemoryException' was thrown.
Error: Failed to download archive 'https://codeload.github.com/<org>/<repo>/tar.gz/<sha>' after 3 attempts.

Runner and Worker's Diagnostic Logs

N/A

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The download path is in src/Runner.Worker/ActionManager.cs around lines 1662-1682; start by reading the GetAsync, ReadAsStreamAsync, and retry flow there. Done means large action archives stream to disk with fixed memory usage, including across retries, without changing successful downloads or extraction behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.