auto-updates fail with `opendir(boot): Operation not permitted` when /boot is a systemd automount that has idled out
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 230
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 38
Description
⚠️ DISCLAIMER. Report is generated by Claude Code while it was troubleshooting one of my bench systems.
I've read the report and indeed seems to be true.
If you don't want to read an LLM report, please stop here. Feel free to close the report.
---
## Summary
`bootc-fetch-apply-updates.service` does not declare `RequiresMountsFor=/boot`. On systems where `/boot` is a `systemd-gpt-auto-generator` automount with `TimeoutIdleSec=120`, `/boot` unmounts ~2 minutes after boot and the update timer then fires against an unmounted `/boot`, failing with:
```
error: Initializing storage: opendir(boot): Operation not permitted
```
The same failure hits `bootc status` and `bootc upgrade` run manually.
ostree hardened its own units against this exact hazard in ostreedev/ostree#2544; bootc's updater unit never got the equivalent.
## Environment
- bootc `1.16.7-1.fc44`
- ostree `2026.3-1.fc44`
- systemd `259.8-1.fc44`
- Base image `quay.io/fedora/fedora-bootc:44`
- BIOS/GRUB boot: `sda1` BIOS boot, `sda2` vfat ESP (unused), `sda3` xfs root
- `/` is a composefs overlay; no `/etc/fstab`
## Symptom
```
Aug 21 18:40:55 systemd[1]: Starting bootc-fetch-apply-updates.service - Apply bootc updates...
Aug 21 18:40:56 bootc[2194]: error: Initializing storage: opendir(boot): Operation not permitted
Aug 21 18:40:56 systemd[1]: bootc-fetch-apply-updates.service: Failed with result 'exit-code'.
```
Repeats on every timer firing. `opendir(...)` is libostree's `glnx_throw_errno_prefix` style, i.e. `EPERM` from `openat(sysroot_fd, "boot")`.
## Reproducer
Deterministic on an affected system:
```console
$ sudo umount /boot # simulate the automount idling out
$ sudo bootc status
error: Status: opendir(boot): Operation not permitted
```
Note bootc's access does **not** trigger the automount — `/boot` remains unmounted afterwards. (I suspect this is because bootc re-execs under `unshare -m` and autofs won't service the request from that private mount namespace, but I haven't confirmed that.)
Touch `/boot` by any other means and bootc works again:
```console
$ ls /boot >/dev/null && sudo bootc status # succeeds
```
## Root cause
Two generators both claim `/boot`:
- `ostree-system-generator` -> `/run/systemd/generator/boot.mount` (`What=/sysroot/boot`, `Options=bind`) — the real mount
- `systemd-gpt-auto-generator` -> `/run/systemd/generator.late/boot.automount` (`Where=/boot`, `TimeoutIdleSec=120`, described as "EFI System Partition Automount") — targeting the otherwise-unused ESP
When the autofs idle timer fires it tears down ostree's `/boot` bind mount. The ostree units survive this because they declare the dependency:
```console
$ systemctl show boot.mount -p RequiredBy
RequiredBy=ostree-finalize-staged.service ostree-boot-complete.service \
ostree-finalize-staged-hold.service rpm-ostree-fix-shadow-mode.service local-fs.target
$ systemctl show bootc-fetch-apply-updates.service -p RequiresMountsFor
RequiresMountsFor=
```
## Why this is newly visible
This system ran fine for two weeks on ostree `2026.2` and broke on upgrade to `2026.3`, with bootc, systemd and the kernel at identical versions across both deployments. The behaviour change is ostreedev/ostree@62109dca ("prepare-root: create `/run/systemd/volatile-root` for composefs"), whose commit message states:
> With composefs enabled both are overlayfs mounts with an anonymous `st_dev` … As a result systemd-gpt-auto-generator silently skips all partition discovery: the ESP is never automounted on /boot
Before 2026.3, gpt-auto could not resolve the composefs root's block device and skipped partition discovery entirely, so `boot.automount` was never generated. Journal evidence — `Set up automount boot.automount` appears only in deployments running 2026.3:
```
boot -3: ostree 2026.2 automount=0
boot -2: ostree 2026.3 automount=1 <- /boot unmounted 2m41s after boot
boot -1: ostree 2026.2 automount=0
boot 0: ostree 2026.3 automount=1
```
So ostree 2026.3 makes gpt-auto work as intended, which in turn exposes the missing dependency in bootc. I'd expect this to affect any composefs-based bootc system with a discoverable ESP as 2026.3 rolls out.
## Proposed fix
Mirror what ostree does — add to `bootc-fetch-apply-updates.service`:
```ini
[Unit]
RequiresMountsFor=/boot
```
Verified as effective here via a drop-in: systemd then pulls in `boot.mount` and orders the service after it.
Two secondary suggestions:
1. `opendir(boot): Operation not permitted` is opaque for what is really "/boot is not mounted". A dedicated check with a clearer message would save a lot of digging.
2. Any other bootc command touching `/boot` (`bootc status`, `bootc upgrade`, `bootc switch`) has the same exposure when run from an idle shell, and can't be fixed by unit dependencies. Triggering the automount explicitly, or reporting a clear error, would help.
## References
- ostreedev/ostree#2544 — ostree's fix for the same hazard on its own units
- ostreedev/ostree@62109dca — the change that makes gpt-auto discover partitions on composefs roots
- systemd/systemd#13099 — gpt-auto generates `boot.automount` despite an existing `boot.mount`
- systemd/systemd#35017 — gpt-auto with overlay roots
Contributor guide
Research direction
Start at the bootc-fetch-apply-updates.service unit and inspect how its systemd dependencies are defined. Verify the service declares the required dependency on /boot, then use systemctl to confirm boot.mount is pulled in and ordered before the service; the reproducer is the unmounted /boot case followed by bootc status or the update service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- devops, operating-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100