bootc-dev / bootc-dev/bootc

auto-updates fail with `opendir(boot): Operation not permitted` when /boot is a systemd automount that has idled out

Offen Anfängerfreundlich
#2,402 9 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
triaged
Vorherrschende Sprache
Rust
Sterne
2.3k
Forks
230
Ø Merge
3 T. 3 Std.
Gemergte PRs (30 T.)
36

Beschreibung

⚠️ DISCLAIMER. Report is generated by Claude Code while it was troubleshooting one of my bench systems.
I've read the report and indeed seems to be true.
If you don't want to read an LLM report, please stop here. Feel free to close the report.

---

## Summary

`bootc-fetch-apply-updates.service` does not declare `RequiresMountsFor=/boot`. On systems where `/boot` is a `systemd-gpt-auto-generator` automount with `TimeoutIdleSec=120`, `/boot` unmounts ~2 minutes after boot and the update timer then fires against an unmounted `/boot`, failing with:

```
error: Initializing storage: opendir(boot): Operation not permitted
```

The same failure hits `bootc status` and `bootc upgrade` run manually.

ostree hardened its own units against this exact hazard in ostreedev/ostree#2544; bootc's updater unit never got the equivalent.

## Environment

- bootc `1.16.7-1.fc44`
- ostree `2026.3-1.fc44`
- systemd `259.8-1.fc44`
- Base image `quay.io/fedora/fedora-bootc:44`
- BIOS/GRUB boot: `sda1` BIOS boot, `sda2` vfat ESP (unused), `sda3` xfs root
- `/` is a composefs overlay; no `/etc/fstab`

## Symptom

```
Aug 21 18:40:55 systemd[1]: Starting bootc-fetch-apply-updates.service - Apply bootc updates...
Aug 21 18:40:56 bootc[2194]: error: Initializing storage: opendir(boot): Operation not permitted
Aug 21 18:40:56 systemd[1]: bootc-fetch-apply-updates.service: Failed with result 'exit-code'.
```

Repeats on every timer firing. `opendir(...)` is libostree's `glnx_throw_errno_prefix` style, i.e. `EPERM` from `openat(sysroot_fd, "boot")`.

## Reproducer

Deterministic on an affected system:

```console
$ sudo umount /boot # simulate the automount idling out
$ sudo bootc status
error: Status: opendir(boot): Operation not permitted
```

Note bootc's access does **not** trigger the automount — `/boot` remains unmounted afterwards. (I suspect this is because bootc re-execs under `unshare -m` and autofs won't service the request from that private mount namespace, but I haven't confirmed that.)

Touch `/boot` by any other means and bootc works again:

```console
$ ls /boot >/dev/null && sudo bootc status # succeeds
```

## Root cause

Two generators both claim `/boot`:

- `ostree-system-generator` -> `/run/systemd/generator/boot.mount` (`What=/sysroot/boot`, `Options=bind`) — the real mount
- `systemd-gpt-auto-generator` -> `/run/systemd/generator.late/boot.automount` (`Where=/boot`, `TimeoutIdleSec=120`, described as "EFI System Partition Automount") — targeting the otherwise-unused ESP

When the autofs idle timer fires it tears down ostree's `/boot` bind mount. The ostree units survive this because they declare the dependency:

```console
$ systemctl show boot.mount -p RequiredBy
RequiredBy=ostree-finalize-staged.service ostree-boot-complete.service \
ostree-finalize-staged-hold.service rpm-ostree-fix-shadow-mode.service local-fs.target

$ systemctl show bootc-fetch-apply-updates.service -p RequiresMountsFor
RequiresMountsFor=
```

## Why this is newly visible

This system ran fine for two weeks on ostree `2026.2` and broke on upgrade to `2026.3`, with bootc, systemd and the kernel at identical versions across both deployments. The behaviour change is ostreedev/ostree@62109dca ("prepare-root: create `/run/systemd/volatile-root` for composefs"), whose commit message states:

> With composefs enabled both are overlayfs mounts with an anonymous `st_dev` … As a result systemd-gpt-auto-generator silently skips all partition discovery: the ESP is never automounted on /boot

Before 2026.3, gpt-auto could not resolve the composefs root's block device and skipped partition discovery entirely, so `boot.automount` was never generated. Journal evidence — `Set up automount boot.automount` appears only in deployments running 2026.3:

```
boot -3: ostree 2026.2 automount=0
boot -2: ostree 2026.3 automount=1 <- /boot unmounted 2m41s after boot
boot -1: ostree 2026.2 automount=0
boot 0: ostree 2026.3 automount=1
```

So ostree 2026.3 makes gpt-auto work as intended, which in turn exposes the missing dependency in bootc. I'd expect this to affect any composefs-based bootc system with a discoverable ESP as 2026.3 rolls out.

## Proposed fix

Mirror what ostree does — add to `bootc-fetch-apply-updates.service`:

```ini
[Unit]
RequiresMountsFor=/boot
```

Verified as effective here via a drop-in: systemd then pulls in `boot.mount` and orders the service after it.

Two secondary suggestions:

1. `opendir(boot): Operation not permitted` is opaque for what is really "/boot is not mounted". A dedicated check with a clearer message would save a lot of digging.
2. Any other bootc command touching `/boot` (`bootc status`, `bootc upgrade`, `bootc switch`) has the same exposure when run from an idle shell, and can't be fixed by unit dependencies. Triggering the automount explicitly, or reporting a clear error, would help.

## References

- ostreedev/ostree#2544 — ostree's fix for the same hazard on its own units
- ostreedev/ostree@62109dca — the change that makes gpt-auto discover partitions on composefs roots
- systemd/systemd#13099 — gpt-auto generates `boot.automount` despite an existing `boot.mount`
- systemd/systemd#35017 — gpt-auto with overlay roots

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie bei der Unit bootc-fetch-apply-updates.service und untersuchen Sie, wie ihre systemd-Abhängigkeiten definiert sind. Überprüfen Sie, dass der Dienst die erforderliche Abhängigkeit von /boot deklariert, und verwenden Sie anschließend systemctl, um zu bestätigen, dass boot.mount einbezogen und vor dem Dienst eingeordnet wird; der Reproducer ist der Fall eines nicht eingehängten /boot, gefolgt von bootc status oder dem Update-Dienst.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
linux, rust
Bereich
devops, operating-systems
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
70/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.