actions / actions/runner

False disk space warning on immutable/composefs root filesystems (Directory.GetDirectoryRoot always returns / on Linux)

Open
#4,380 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Description

Self-hosted runners on immutable Linux systems (bootc, Fedora CoreOS, ostree-based) report a false disk space warning every job:

Warning: You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB

There is plenty of real disk space available. The warning fires because of a platform-specific bug in CheckDiskSpaceAsync.

Root Cause

In src/Runner.Worker/JobExtension.cs, the disk space check does:

var workDirRoot = Directory.GetDirectoryRoot(HostContext.GetDirectory(WellKnownDirectory.Work));
var driveInfo = new DriveInfo(workDirRoot);
var freeSpaceInMB = driveInfo.AvailableFreeSpace / 1024 / 1024;

On Linux, Directory.GetDirectoryRoot() always returns / regardless of which filesystem the work directory is actually mounted on. This means DriveInfo always inspects / — not the filesystem backing the work directory.

On immutable systems, / is a composefs mount (the read-only image layer). composefs correctly reports 0 bytes available because it is a read-only filesystem. The runner interprets this as the disk being full.

Environment

  • Runner version: 2.334.0
  • OS: Fedora CoreOS / MethidOS (bootc-managed image)
  • Root filesystem: composefs (immutable, read-only — this is by design)
  • Actual writable partitions: hundreds of GB free (verified with df -h)

Example df -h output from an affected machine:

Filesystem      Size  Used Avail Use% Mounted on
composefs        47M   47M     0 100% /              ← always 0, read-only by design
/dev/nvme0n1p5  141G   24G  110G  18% /etc
/dev/nvme0n1p3  619G  106G  514G  17% /var/lib/docker
/dev/nvme0n1p4  186G    2G  175G   2% /var/home

The runner work directory (/var/lib/github-runner/_work) is on /dev/nvme0n1p5 with 110 GB free. GetDirectoryRoot returns / for this path, causing DriveInfo to check composefs instead.

Expected Behavior

The disk space check should inspect the filesystem that actually backs the work directory, not always /.

Suggested Fix

Replace Directory.GetDirectoryRoot() + DriveInfo with a direct statvfs(2) call (via P/Invoke or DriveInfo on the full work path) on Linux, or use the df-equivalent .NET API that resolves the actual mount point for a given path:

// Instead of:
var workDirRoot = Directory.GetDirectoryRoot(HostContext.GetDirectory(WellKnownDirectory.Work));
var driveInfo = new DriveInfo(workDirRoot);

// Use the work path directly — DriveInfo accepts a directory path, not just a root:
var driveInfo = new DriveInfo(HostContext.GetDirectory(WellKnownDirectory.Work));

new DriveInfo(path) on .NET resolves the actual volume for the given path on Windows, but on Linux it still resolves to /. The correct Linux fix is to call statvfs directly on the work directory path so the kernel returns stats for the actual mount.

Additional Context

This affects any self-hosted runner on an ostree/bootc/composefs-based OS, including Fedora CoreOS (41+), Universal Blue images, and any custom bootc image. These systems are increasingly common for edge/embedded CI workloads. The warning is purely cosmetic today but causes user confusion and will become a hard failure if the runner ever starts refusing jobs based on this check.

There is no runner-side configuration knob to suppress or redirect this check — system.runner.lowdiskspacethreshold is pushed by GitHub's service in the job payload and cannot be overridden from the runner's .env file.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read src/Runner.Worker/JobExtension.cs and trace CheckDiskSpaceAsync, starting with how it derives the work-directory filesystem and calculates available space. Verify the change against a composefs root with a writable work-directory mount; done means the warning reports the work filesystem's available space instead of the read-only root filesystem.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, linux
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.