actions / actions/runner

Sometimes /mnt (/dev/sdb1) is 100% full when the build job starts

Open
#3,968 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug

In very rare cases, the /mnt disk (/dev/sdb1) is 100% full.
This happens very rarely. Example of such case:
https://github.com/apache/pulsar/actions/runs/16641732202/job/47093641342?pr=24593#step:3:118

  Filesystem     1M-blocks   Used Available Use% Mounted on
  /dev/root         74245M 52119M    22111M  71% /
  tmpfs              7998M     1M     7998M   1% /dev/shm
  tmpfs              3200M     2M     3199M   1% /run
  tmpfs                 5M     0M        5M   0% /run/lock
  /dev/sda15          105M     7M       99M   6% /boot/efi
  /dev/sdb1         75029M 73225M        0M 100% /mnt
  tmpfs              1600M     1M     1600M   1% /run/user/1001

To Reproduce

There isn't a way to reproduce this. It happens rarely. In apache/pulsar actions, I've seen this a few times in the last couple of months.

Expected behavior

/mnt disk should have plenty of available disk space

Runner Version and Platform

Example of where /mnt is 100% full: https://github.com/apache/pulsar/actions/runs/16641732202/job/47093641342?pr=24593#step:3:118

Current runner version: '2.327.1'
Runner Image Provisioner
  Hosted Compute Agent
  Version: 20250711.363
  Commit: 6785254374ce925a23743850c1cb91912ce5c14c
  Build Date: 2025-07-11T20:04:25Z
Operating System
  Ubuntu
  22.04.5
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20250728.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20250728.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20250728.1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Apache Pulsar job log and compare its runner version, Ubuntu image, and /mnt usage with other affected runs. Trace the hosted runner provisioning context from those details; done means identifying a validated cause for the intermittent full /mnt disk and defining a runner-side correction or mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.