Btrfs root filesystem reached 0 unallocated space mid-`pacman -Syu`, corrupting kernel package and breaking boot

## Summary

An interrupted `pacman -Syu` transaction left the system unbootable: `/boot` was missing all kernel/initramfs files (while pacman's local database still recorded the kernels as installed), and the local package database entry for `linux-cachyos-lts` was corrupted (only `mtree` survived, `desc`/`files` were gone). Root cause: the Btrfs root filesystem had gradually reached **0 unallocated device space** (all 465.26 GiB allocated to chunks, despite `df` reporting ~57 GiB "free"), and a chunk allocation failure mid-transaction corrupted state with no clear error surfaced to the user.

No `btrfs-balance` timer was enabled by default, so nothing periodically reclaimed unused space from allocated chunks before the filesystem ran out of allocatable headroom entirely.

## Environment

- CachyOS rolling, `linux-cachyos 7.2.4-1`, `linux-cachyos-lts 6.18.50-1`
- Btrfs root (`@` subvolume), single NVMe device, 465.26 GiB
- Snapper enabled with default-ish config (`NUMBER_LIMIT=50`, `TIMELINE_CREATE=no`)
- KDE Plasma desktop

## Timeline (reconstructed from `journalctl -b -2`)

| Time (Sep 11) | Event |
|---|---|
| 16:23:37 | `pacman --color always -Syu` launched by user |
| 16:24:31 | Snapper pre-transaction snapshot **55** created |
| 16:24:33 | gpg-agent started (pacman keyring check) |
| — | *(transaction proceeds, writing packages including `linux-cachyos-lts`)* |
| 16:44:20 | **First ENOSPC symptom**: KDE fails to write `~/.config/fcitx5/profile` — "No space left on device" |
| 16:51–00:45 | Cascading ENOSPC failures across unrelated services: NetworkManager (`resolv.conf`), bluetoothd, mandb, wireplumber, Steam symlink creation |
| — | **No corresponding post-transaction snapshot 56 was ever created** — the transaction never completed |
| Next boot | GRUB error: `you need to load the kernel first` — `/boot` contains no `vmlinuz-*` or `initramfs-*` files |

## Root cause: Btrfs allocation exhaustion

From a live-USB chroot investigation:

```
Device size: 465.26 GiB
Device allocated: 465.26 GiB
Device unallocated: 1.05 MiB <-- essentially zero
Used: 407.38 GiB
Free (estimated): 56.86 GiB <-- misleading; this is free space INSIDE
already-allocated chunks, not usable
for new chunk allocation
```

This explains every downstream symptom:
- `cp /etc/resolv.conf` into the chroot failed with ENOSPC despite `df` showing 57 GiB free
- `pacman -Syy` failed to create its temporary download directory with ENOSPC
- Conservative `btrfs balance start -dusage=5` / `-musage=50` both failed with ENOSPC (no free unallocated space to relocate chunks into)
- Only `-dusage=0` / `-musage=0` "succeeded," trivially, because no chunk was 100% empty

**`df`/`statfs` free-space reporting is not a reliable signal of Btrfs allocation health**, and nothing in the stock CachyOS install monitored or alerted on this distinction.

## Casualty: `linux-cachyos-lts`

The local pacman database entry was left corrupted:
```
$ ls -la /mnt/var/lib/pacman/local/linux-cachyos-lts-6.18.50-1/
-rw-r--r-- 1 root root 352504 Sep 10 19:35 mtree
```
Only `mtree` survived; `desc`, `files`, and other metadata were missing. This is consistent with a write that died partway through pacman's local-db update sequence during the chunk-allocation failure. Actual kernel module files on disk were intact (`pacman -Dk` flagged them as "exists in filesystem" when attempting a clean reinstall), so this was a metadata-layer corruption, not data loss.

A secondary, apparently unrelated dependency mismatch (`systemd 261.2` vs installed packages requiring it) was also present post-recovery, resolved by a full `pacman -Syu` once filesystem headroom was restored.

## Recovery steps taken (for reference)

1. Booted CachyOS live USB, mounted root + EFI, chrooted
2. Temporarily added a spare USB block device via `btrfs device add` to obtain unallocated space
3. Ran `btrfs balance start -dusage=20` / `-musage=20`, successfully relocating 41/466 and 2/426 chunks
4. Removed the temporary device (`btrfs device remove`), recovering ~40 GiB unallocated on the primary device
5. Fixed `pacman-key --init && pacman-key --populate` (unrelated GPGME error also present in the chroot)
6. Reinstalled `linux-cachyos` and force-reinstalled `linux-cachyos-lts --overwrite='*'` (required due to the corrupted local db entry)
7. Ran `mkinitcpio -P`, regenerated GRUB config, `grub-install`
8. Full `pacman -Syu` to resolve the secondary `systemd` dependency mismatch
9. `pacman -Dk` returned clean; system now boots normally

## Ask / suggestion

1. **Enable a `btrfs-balance` timer by default** (or at least prompt users to during install) — `btrfsmaintenance` is already packaged and available, but `btrfs-balance.timer` is `disabled` out of the box, and the default thresholds (`BTRFS_BALANCE_DUSAGE="5 10"`, `BTRFS_BALANCE_MUSAGE="5"`) are too conservative to meaningfully reclaim space in a filesystem approaching full allocation (mine was at 96%+ data-chunk usage before this failure).
2. **Consider a pre-transaction sanity check** in pacman hooks (or a CachyOS-specific hook) that checks `btrfs filesystem usage` unallocated space specifically, not just `df` free space, and warns clearly before a large transaction (e.g., kernel installs) if headroom is critically low.
3. At minimum, **surface a clearer error** when a pacman transaction hits Btrfs ENOSPC mid-write, rather than allowing it to silently leave `/boot` and the local package database in an inconsistent state.

Happy to provide additional logs/output if useful.

---

## AI use disclosure

This report was compiled with the assistance of an AI assistant (Claude), which helped diagnose the issue in real time (live-USB troubleshooting session), reconstruct the timeline from `journalctl` output, and draft this write-up. All commands were run and all log output was captured directly from my own system; the AI helped interpret the evidence and organize it into this report.

Start by reviewing the btrfs-balance.timer configuration, the available pacman hook mechanism, and the reported btrfs filesystem usage and ENOSPC behavior. The issue is done when the project decides on a concrete mitigation, implements it, and verifies that low unallocated Btrfs space produces safe recovery or a clear warning before package transactions can leave the system inconsistent.

Written by the indexing model from the issue text.