cosa buildextend-live step: gf-mksquashfs fails to create rootfs squashfs: No space left on device

Bug Report

Environment

What operating system is being used to run coreos-assembler?
Fedora 36/37

What operating system is being assembled?
Custom Fedora CoreOS derivative

Is coreos-assembler running in Podman or Docker?
Podman

If Podman, is coreos-assembler running privileged or unprivileged?
privileged

Expected Behavior

Running cosa buildextend-live generates a live iso installer image

Actual Behavior

In the cosa buildextend-live phase, during squashfs generation for the rootfs, the gf-mkquashfs command fails due to No space left on Device even though there is space available on the mounted volumes in the cosa container.

Our custom FCOS derivative pulls in a large amount of rpm packages (~5.3GB worth). It seems that we are hitting a hard size limit for how big the live-iso can grow to be. If we reduce the number of rpm packages included in the manifest then the live iso generation step runs fine.

Reproduction Steps

1. cosa init
2. cosa fetch
3. cosa build
4. cosa buildextend-metal
5. cosa buildextend-metal4k
6. cosa buildextend-live < fails here

Other Information

Build output:

++ guestfish --remote -- list-partitions
+ partitions='/dev/sda1
/dev/sda2
/dev/sda3
/dev/sda4'
+ for pt in $partitions
++ coreos_gf vfs-label /dev/sda1
++ guestfish --remote -- vfs-label /dev/sda1
+ label=
+ '[' '' == EFI-SYSTEM ']'
+ for pt in $partitions
++ coreos_gf vfs-label /dev/sda2
++ guestfish --remote -- vfs-label /dev/sda2
+ label=EFI-SYSTEM
+ '[' EFI-SYSTEM == EFI-SYSTEM ']'
+ coreos_gf mount /dev/sda2 /boot/efi
+ guestfish --remote -- mount /dev/sda2 /boot/efi
+ for pt in $partitions
++ coreos_gf vfs-label /dev/sda3
++ guestfish --remote -- vfs-label /dev/sda3
+ label=boot
+ '[' boot == EFI-SYSTEM ']'
+ for pt in $partitions
++ coreos_gf vfs-label /dev/sda4
++ guestfish --remote -- vfs-label /dev/sda4
+ label=root
+ '[' root == EFI-SYSTEM ']'
++ coreos_gf ls /ostree/deploy
++ guestfish --remote -- ls /ostree/deploy
+ stateroot=/ostree/deploy/fedora-coreos
++ coreos_gf ls /ostree/deploy/fedora-coreos/deploy
++ guestfish --remote -- ls /ostree/deploy/fedora-coreos/deploy
++ grep -v .origin
+ deploydir=/ostree/deploy/fedora-coreos/deploy/585760e354d85005f83fb2413f96f6fe0a0d758241ef0a44073e8c8d4446602d.0
+ export stateroot deploydir
+ coreos_gf download /ostree/repo/config /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/config
+ guestfish --remote -- download /ostree/repo/config /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/config
+ grep -v readonly=true /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/config
+ coreos_gf upload /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/config.new /ostree/repo/config
+ guestfish --remote -- upload /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/config.new /ostree/repo/config
+ coreos_gf mksquashfs / /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/image.squashfs compress:zstd
+ guestfish --remote -- mksquashfs / /srv/tmp/buildpost-live/initrd-rootfs/gf-mksquashfs.UHLaDw/image.squashfs compress:zstd
libguestfs: error: mksquashfs: /: Write failed because No space left on device
FATAL ERROR: Failed to write to output filesystem
+ _coreos_gf_cleanup
+ guestfish --remote -- exit
+ :
Error running command /usr/lib/coreos-assembler/gf-mksquashfs

with a more recent traceback:

2022-11-16 20:50:47,681 ERROR - COMMAND: ['/usr/lib/coreos-assembler/gf-mksquashfs', 'builds/36.20221116.dev.0/x86_64/fedora-coreos-36.20221116.dev.0-metal.x86_64.raw', '/srv/tmp/buildpost-live/initrd-rootfs/root.squashfs', 'zstd']
Traceback (most recent call last):
  File "/usr/lib/coreos-assembler/cmd-buildextend-live", line 738, in <module>
    generate_iso()
  File "/usr/lib/coreos-assembler/cmd-buildextend-live", line 331, in generate_iso
    runcmd(['/usr/lib/coreos-assembler/gf-mksquashfs',
  File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 78, in runcmd
    raise e
  File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 70, in runcmd
    cp = subprocess.run(cmd, **pargs)
  File "/usr/lib64/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/lib/coreos-assembler/gf-mksquashfs', 'builds/36.20221116.dev.0/x86_64/fedora-coreos-36.20221116.dev.0-metal.x86_64.raw', '/srv/tmp/buildpost-live/initrd-rootfs/root.squashfs', 'zstd']' returned non-zero exit status 1.

Start with cmd-buildextend-live around generate_iso() and line 331, then inspect /usr/lib/coreos-assembler/gf-mksquashfs and the guestfish mksquashfs invocation. Reproduce the failure with the listed cosa steps and determine which filesystem limit is exhausted; done means large custom images complete successfully or the failing capacity is identified and reported clearly.