bootc status/upgrade fails after upgrade with ostree-finalize-staged.service exceeding timeout, rendering bootc unusable
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 230
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 38
Description
I have been enjoying using Fedora bootc for quite a while now in my homelab :)
Occasionally after updating one of my systems (image based on fedora-bootc:42) I get this error message from bootc upgrade (and bootc status):
```shell
$ sudo bootc upgrade
error: Upgrading: Computing status: Rollback deployment: Reading deployment metadata: Missing base image ref ostree/container/blob/sha256_3A_0b11f38bc7a328029152915418e3d218669bacef4aafafd79e6501d7c4622e9b
```
It seems the missing ref is always from the rollback image and the error occurs if the `ostree-finalize-staged.service` times out after a previous upgrade. The default timeout is set to 5min:
```ini
# /usr/lib/systemd/system/ostree-finalize-staged.service
[Service]
# This is a quite long timeout intentionally; the failure mode
# here is that people don't get an upgrade. We need to handle
# cases with slow rotational media, etc.
TimeoutStopSec=5m
```
Evidently this is not enough for my arm SBC (librecomputer lafrite) with slow USB storage. After overwriting it with `TimeoutStopSec=10m` I have not encountered this issue for 3 months.
But unlike what the comment in the service file suggests is does not lead to the upgrade failing, but in **rendering the system unable to upgade, rollback or applying another image**.
Unfortunately I fond no documentation describing how to recover from this error (only #219, seems related?). Instead I managed to recover my system using the undocumented internals commands, but is there a better/intended way?
```shell
$ sudo bootc upgrade
error: Upgrading: Computing status: Rollback deployment: Reading deployment metadata: Missing base image ref ostree/container/blob/sha256_3A_0b11f38bc7a328029152915418e3d218669bacef4aafafd79e6501d7c4622e9b
$ sudo mount -o remount,rw /sysroot
# pull the missing (?) rollback image
$ sudo bootc internals ostree-ext container image pull /sysroot/ostree/repo ostree-unverified-registry:registry.local/bootc/lafrite-bootc@sha256:a3318fb5cd371288b77a64f3177e9b489e7c2b4f1e357db769e27605462ad880
layers already present: 24; layers needed: 42 (593.2 MB)
43.56 MiB [████████████████████] (0s) Fetched layer fb35d93796e742807a8 Image contains non-ostree compatible file paths: dnf: 1 lock: 2 nvim.root: 1 systemd: 3
Wrote: ostree-unverified-registry:registry.local/bootc/lafrite-bootc@sha256:a3318fb5cd371288b77a64f3177e9b489e7c2b4f1e357db769e27605462ad880 => 03d623e3219f159d0a07623520bb6da1c8dfc2364101f238ea5f29cf4131386f
[trisse@lafrite ~]$ sudo mount -o remount,ro /sysroot
[trisse@lafrite ~]$ sudo bootc status
● Booted image: registry.local/bootc/lafrite-bootc:42
Digest: sha256:4c6b01b2bd541b1352048ef2d4f58fd9b09f9574cc9f3de533287d039d735b97 (arm64)
Version: 42.20250911.0 (2025-09-12T07:09:00Z)
Rollback image: registry.local/bootc/lafrite-bootc:42
Digest: sha256:a3318fb5cd371288b77a64f3177e9b489e7c2b4f1e357db769e27605462ad880 (arm64)
Version: 42.20250904.0 (2025-09-05T07:09:40Z)
```
Thanks in advance!
Contributor guide
Assessment
This issue has not been assessed yet.