coreos / coreos/coreos-assembler

osbuild flaking ENOSPC with after build-with-buildah enabled in rawhide

Open
#4,289 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
393
Forks
193
Avg merge
17h 6m
Merged PRs (30d)
5

Description

I've noticed a few things flaking runing out of space in the created disk image inside OSBuild

The disk size estimation code seems to be in line with recent non build-with-buildah builds:

link to failed run

08:10:28  Estimating disk size...
08:10:29  Disk sizes: metal: 2839M (estimated), cloud: 10240M

but then we fail in OSBuild:

[2025-08-27T12:15:44.263Z] org.osbuild.copy: e00e83481f729b32f008f561a487906c2de9f4cd0e98e5f1defa6b4d2d0930da {
[2025-08-27T12:15:44.263Z]   "paths": [
[2025-08-27T12:15:44.263Z]     {
[2025-08-27T12:15:44.263Z]       "from": "input://tree/",
[2025-08-27T12:15:44.263Z]       "to": "mount://root/"
[2025-08-27T12:15:44.263Z]     }
[2025-08-27T12:15:44.263Z]   ]
[2025-08-27T12:15:44.263Z] }
[2025-08-27T12:15:44.263Z] device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
[2025-08-27T12:15:44.263Z] mount/root (org.osbuild.xfs): mounting /dev/loop0p4 -> /home/jenkins/agent/workspace/test-override/cache/osbuild/store/tmp/buildroot-tmp-u0z210wp/mounts/
[2025-08-27T12:15:44.516Z] mount/boot (org.osbuild.ext4): mounting /dev/loop0p3 -> /home/jenkins/agent/workspace/test-override/cache/osbuild/store/tmp/buildroot-tmp-u0z210wp/mounts/boot
[2025-08-27T12:15:44.516Z] mount/efi (org.osbuild.fat): mounting /dev/loop0p2 -> /home/jenkins/agent/workspace/test-override/cache/osbuild/store/tmp/buildroot-tmp-u0z210wp/mounts/boot/efi
[2025-08-27T12:15:44.516Z] Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
[2025-08-27T12:15:44.769Z] copying '/run/osbuild/inputs/tree/.' -> '/run/osbuild/mounts/.'
[2025-08-27T12:15:51.285Z] 
[2025-08-27T12:15:51.285Z] Filesystem      Size  Used Avail Use% Mounted on
[2025-08-27T12:15:51.285Z] /dev/vdb1        45G  9.5G   36G  22% /home/jenkins/agent/workspace/test-override/cache
[2025-08-27T12:15:51.285Z] 
[2025-08-27T12:16:01.203Z] cp: cannot create symbolic link '/run/osbuild/mounts/././ostree/repo/objects/68/2244a3557ca80c49194eb34ea0270ee803ccea61a4c623c07e3b8da77934e6.file': No space left on device

log.txt

A subsequent run passes in this step. I'm not sure what is going on here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Jenkins failed run and log.txt, then compare the failing and subsequent passing runs. Trace the disk size estimation and the OSBuild step with build-with-buildah enabled, focusing on the reported ENOSPC during the ostree copy. Done means identifying the cause of the flake and preventing the created image from running out of space.

Written by the indexing model from the issue text.

Assessment

Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.