bootc-dev / bootc-dev/bootc

auto-updates fail with `opendir(boot): Operation not permitted` when /boot is a systemd automount that has idled out

Open Beginner friendly
#2,402 9 comments 1 reaction 0 assignees View on GitHub
triaged
Dominant language
Rust
Stars
2.3k
Forks
230
Avg merge
3d 12h
Merged PRs (30d)
38

Description

⚠️ DISCLAIMER. Report is generated by Claude Code while it was troubleshooting one of my bench systems.
I've read the report and indeed seems to be true.
If you don't want to read an LLM report, please stop here. Feel free to close the report.

---

## Summary

`bootc-fetch-apply-updates.service` does not declare `RequiresMountsFor=/boot`. On systems where `/boot` is a `systemd-gpt-auto-generator` automount with `TimeoutIdleSec=120`, `/boot` unmounts ~2 minutes after boot and the update timer then fires against an unmounted `/boot`, failing with:

```
error: Initializing storage: opendir(boot): Operation not permitted
```

The same failure hits `bootc status` and `bootc upgrade` run manually.

ostree hardened its own units against this exact hazard in ostreedev/ostree#2544; bootc's updater unit never got the equivalent.

## Environment

- bootc `1.16.7-1.fc44`
- ostree `2026.3-1.fc44`
- systemd `259.8-1.fc44`
- Base image `quay.io/fedora/fedora-bootc:44`
- BIOS/GRUB boot: `sda1` BIOS boot, `sda2` vfat ESP (unused), `sda3` xfs root
- `/` is a composefs overlay; no `/etc/fstab`

## Symptom

```
Aug 21 18:40:55 systemd[1]: Starting bootc-fetch-apply-updates.service - Apply bootc updates...
Aug 21 18:40:56 bootc[2194]: error: Initializing storage: opendir(boot): Operation not permitted
Aug 21 18:40:56 systemd[1]: bootc-fetch-apply-updates.service: Failed with result 'exit-code'.
```

Repeats on every timer firing. `opendir(...)` is libostree's `glnx_throw_errno_prefix` style, i.e. `EPERM` from `openat(sysroot_fd, "boot")`.

## Reproducer

Deterministic on an affected system:

```console
$ sudo umount /boot # simulate the automount idling out
$ sudo bootc status
error: Status: opendir(boot): Operation not permitted
```

Note bootc's access does **not** trigger the automount — `/boot` remains unmounted afterwards. (I suspect this is because bootc re-execs under `unshare -m` and autofs won't service the request from that private mount namespace, but I haven't confirmed that.)

Touch `/boot` by any other means and bootc works again:

```console
$ ls /boot >/dev/null && sudo bootc status # succeeds
```

## Root cause

Two generators both claim `/boot`:

- `ostree-system-generator` -> `/run/systemd/generator/boot.mount` (`What=/sysroot/boot`, `Options=bind`) — the real mount
- `systemd-gpt-auto-generator` -> `/run/systemd/generator.late/boot.automount` (`Where=/boot`, `TimeoutIdleSec=120`, described as "EFI System Partition Automount") — targeting the otherwise-unused ESP

When the autofs idle timer fires it tears down ostree's `/boot` bind mount. The ostree units survive this because they declare the dependency:

```console
$ systemctl show boot.mount -p RequiredBy
RequiredBy=ostree-finalize-staged.service ostree-boot-complete.service \
ostree-finalize-staged-hold.service rpm-ostree-fix-shadow-mode.service local-fs.target

$ systemctl show bootc-fetch-apply-updates.service -p RequiresMountsFor
RequiresMountsFor=
```

## Why this is newly visible

This system ran fine for two weeks on ostree `2026.2` and broke on upgrade to `2026.3`, with bootc, systemd and the kernel at identical versions across both deployments. The behaviour change is ostreedev/ostree@62109dca ("prepare-root: create `/run/systemd/volatile-root` for composefs"), whose commit message states:

> With composefs enabled both are overlayfs mounts with an anonymous `st_dev` … As a result systemd-gpt-auto-generator silently skips all partition discovery: the ESP is never automounted on /boot

Before 2026.3, gpt-auto could not resolve the composefs root's block device and skipped partition discovery entirely, so `boot.automount` was never generated. Journal evidence — `Set up automount boot.automount` appears only in deployments running 2026.3:

```
boot -3: ostree 2026.2 automount=0
boot -2: ostree 2026.3 automount=1 <- /boot unmounted 2m41s after boot
boot -1: ostree 2026.2 automount=0
boot 0: ostree 2026.3 automount=1
```

So ostree 2026.3 makes gpt-auto work as intended, which in turn exposes the missing dependency in bootc. I'd expect this to affect any composefs-based bootc system with a discoverable ESP as 2026.3 rolls out.

## Proposed fix

Mirror what ostree does — add to `bootc-fetch-apply-updates.service`:

```ini
[Unit]
RequiresMountsFor=/boot
```

Verified as effective here via a drop-in: systemd then pulls in `boot.mount` and orders the service after it.

Two secondary suggestions:

1. `opendir(boot): Operation not permitted` is opaque for what is really "/boot is not mounted". A dedicated check with a clearer message would save a lot of digging.
2. Any other bootc command touching `/boot` (`bootc status`, `bootc upgrade`, `bootc switch`) has the same exposure when run from an idle shell, and can't be fixed by unit dependencies. Triggering the automount explicitly, or reporting a clear error, would help.

## References

- ostreedev/ostree#2544 — ostree's fix for the same hazard on its own units
- ostreedev/ostree@62109dca — the change that makes gpt-auto discover partitions on composefs roots
- systemd/systemd#13099 — gpt-auto generates `boot.automount` despite an existing `boot.mount`
- systemd/systemd#35017 — gpt-auto with overlay roots

Contributor guide

Open the contributing guide

Research direction

Start at the bootc-fetch-apply-updates.service unit and inspect how its systemd dependencies are defined. Verify the service declares the required dependency on /boot, then use systemctl to confirm boot.mount is pulled in and ordered before the service; the reproducer is the unmounted /boot case followed by bootc status or the update service.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
devops, operating-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.