install: Add support for preserving system state during install to-existing-root.
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 230
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 38
Description
The following is an AI assisted spec that describes high level CLI changes to add support for converting a "package mode" system into an "image mode" (bootc) system while preserving system state and configuration.
---
# Design: Package-mode to Image-mode Migration via `bootc install to-existing-root`
This document describes the new migration flags added to
`bootc install to-existing-root` to support converting a running
package-mode (RPM/DEB) system to a bootc image-mode deployment in place,
without a reinstall.
---
## Overview
`bootc install to-existing-root` runs the bootc install pipeline against
the host's root filesystem, replacing the running OS with a new bootc
deployment on the next reboot. By default this is a clean install: the
new deployment's `/var` is empty and `/etc` comes entirely from the image.
Three new flags extend this command for the migration use case:
| Flag | Purpose |
|---|---|
| `--preserve-var` | Copy `/var` data into the new deployment |
| `--merge-etc` | Carry forward `/etc` customisations via 3-way merge |
| `--pkgmode-rollback` | Write a boot entry to roll back to the old OS |
These flags are independent and can be combined freely. The typical
migration invocation is:
```bash
podman run --rm --privileged --pid=host \
-v /:/target \
-v /var/lib/containers:/var/lib/containers \
--security-opt label=type:unconfined_t \
\
bootc install to-existing-root \
--preserve-var \
--merge-etc \
--pkgmode-rollback \
--acknowledge-destructive
```
---
## Flag: `--preserve-var`
After a plain `bootc install to-existing-root`, the new deployment's
`/var` is initially empty — ostree bind-mounts it from a fresh directory
under the stateroot. The package-mode `/var` (databases, application
data, home directories, etc.) is stranded and inaccessible to the new
system.
`--preserve-var` copies `/var` into the new deployment's stateroot
`var/` directory after the core install completes, using one of two
strategies:
- **Reflink copy** (btrfs / XFS): `cp --reflink=always` performs an
instantaneous copy-on-write clone. No extra disk space is consumed
until data diverges. This is probed at runtime.
- **Plain copy** (ext4 and other non-reflink filesystems): `cp -a`
copies each subdirectory of `/var` into the new deployment's stateroot.
Correct but slow for large `/var` trees.
### Rejected alternative: `var.mount` bind-mount unit
An earlier approach injected a `var.mount` systemd unit into the new
deployment's `/etc/systemd/system/` that would bind-mount the old
`/sysroot/var` onto `/var` at boot, avoiding the copy entirely. This was
rejected because both the ostree and composefs backends already
bind-mount the stateroot `var/` onto `/var` in the initramfs, before
systemd starts. A `var.mount` unit fires too late — the mount point is
already in use — and is either silently ignored or fails. The correct
approach is to populate the stateroot `var/` directly before the first
boot, which is what the copy strategies above do.
In both strategies, certain subdirectories are always skipped:
| Skipped path | Reason |
|---|---|
| `var/tmp`, `var/cache` | Ephemeral, regenerated automatically on first boot |
| `var/lib/containers` | Overlay mounts cannot be reflinked or copied across device boundaries |
Additional directories can be excluded via a `--preserve-var-exclude=`
flag. This allows operators to skip large or sensitive directories on a
case-by-case basis — for example, `--preserve-var-exclude=var/log/journal`
on systems with large journals, or `--preserve-var-exclude=var/lib/pgsql`
when the database will be migrated separately.
### Known limitation — live databases on non-reflink filesystems
The plain copy is **unsafe for applications with open write handles**
(PostgreSQL, MySQL, SQLite). Copying a live database produces a corrupted
copy. On ext4 systems, operators must stop stateful services before
running `--preserve-var`, or use a filesystem that supports reflinks
(btrfs, XFS) to take advantage of the atomic copy path. High-risk
directories can be skipped via `--preserve-var-exclude`.
---
## Flag: `--merge-etc`
A plain install populates the new deployment's `/etc` from the image.
The running system's admin customisations — NIC profiles, SSH host
keys, custom CA certificates, application secrets, sysctl drop-ins, etc.
— are not applied to the new deployment.
`--merge-etc` runs a 3-way merge using bootc's existing `etc-merge`
crate after the core install completes. This is the same algorithm
bootc uses during `bootc upgrade`, applied at install time rather than
only at upgrade time:
| Input | Source | Role |
|---|---|---|
| A — pristine baseline | `/usr/etc` | Image's shipped defaults |
| B — current live | `/etc` | Running admin customisations |
| C — new deployment | `/etc` | Merge target |
The diff A→B captures every change the admin made relative to the image
defaults and applies those changes onto C.
Machine-specific files (SSH host keys, machine-id, NIC profiles) are
intentionally included — they are precisely what must be transferred to
make the migrated system functional on first boot.
### Interaction with ostree's /etc semantics
`bootc upgrade` performs this same 3-way merge at upgrade time.
`--merge-etc` at migration time seeds that mechanism correctly, so
subsequent upgrades inherit the right baseline.
---
## Flag: `--pkgmode-rollback`
When set, a "Previous OS" boot entry is written after the install so the
operator can boot back to the old package-mode system if the migration
fails or an application breaks.
The entry is a standard BLS snippet written into the active
`boot/loader.N/entries/` directory alongside the ostree BLS entries:
```
title Previous OS — package-mode rollback (kernel )
sort-key zz-pkgmode
linux /boot/pkgmode-rollback/vmlinuz
initrd /boot/pkgmode-rollback/initramfs.img
options
```
The `zz-pkgmode` sort-key places the entry last in the boot menu.
The `pkgmode-rollback.conf` filename is invisible to ostree (which only
reads `ostree-*.conf` entries) and to bootupd (which does not manage
`loader/entries/`), so it will not be removed by subsequent
`bootc upgrade` operations until explicitly deleted by the operator.
The kernel and initramfs are stashed to `var/lib/pkgmode-rollback/`
before the install wipes `/boot`, then installed to
`boot/pkgmode-rollback/` after the install recreates `/boot`. The
stash lives under `/var` so it survives the `/boot` wipe.
---
## Execution order
The migration steps interlock tightly with the core install pipeline:
```
1. --pkgmode-rollback / --preserve-var:
save_pkgmode_kernel() ← BEFORE install wipes /boot
2. install_to_filesystem() ← core ostree deploy + bootupd
3. --preserve-var:
preserve_var() ← copy /var into new stateroot
4. --pkgmode-rollback:
write_pkgmode_rollback_entry() ← write BLS entry + install kernel
5. --merge-etc:
merge_etc_into_deployment() ← 3-way /etc merge
```
Step 1 must run before step 2. Steps 3–5 are independent of each other
and run after the core install completes.
---
## Interaction with other install options and backends
### composefs backend (`--composefs-backend`)
The composefs backend uses a different on-disk layout from the ostree backend,
and `--preserve-var` and `--merge-etc` must handle both.
**Detection**: after the install completes, the active backend is identified
by which directories are present under ``:
- ostree backend: `/ostree/deploy//deploy//`
- composefs backend: `/composefs/` and `/state/deploy//`
**`--preserve-var`**: the ostree backend gives each stateroot its own
`var/` directory at `/ostree/deploy//var/`. The composefs
backend instead uses a single shared directory at `/state/os/default/var`,
with each deployment's `var` entry being a symlink to it. For the composefs
case, `--preserve-var` must populate `/state/os/default/var` directly
rather than a per-deployment path.
**`--merge-etc`**: the ostree backend stores the deployment's mutable `etc/`
under `/ostree/deploy//deploy//etc/`. In the composefs
backend it is at `/state/deploy//etc/`. The most recently
created entry under `/state/deploy/` identifies the new deployment, the
same heuristic used for the ostree backend.
The pristine baseline (`usr/etc`) is the same in both cases: it is always
read from the deployment's read-only root image, which in the composefs backend
is an EROFS image mounted via composefs. The `/usr/etc` path within that image
is accessible via the deployment's EROFS mount at install time.
The composefs "sealed" variant (UKI + fsverity) embeds a SHA-512 digest
of the root filesystem in the kernel command line. The rollback BLS entry
uses the original package-mode kernel and is unaffected by this; the
stash/restore of vmlinuz/initramfs for `--pkgmode-rollback` is
backend-agnostic.
### Bootloader variants
`--pkgmode-rollback` writes the BLS entry into the same
`boot/loader.N/entries/` directory used by bootupd, and is therefore
compatible with both GRUB and systemd-boot.
When `--bootloader=none` is used, the entry file is still written but
whether it appears in the boot menu depends on the external bootloader
configuration.
### `to-disk` and `to-filesystem`
These three flags are only available on `to-existing-root`. They are
not applicable to `to-disk` or `to-filesystem`, which install onto a
blank or separately mounted target with no running system state to
preserve.
### `--replace=wipe`
`--replace=wipe` wipes the existing root before the install, which is
incompatible in intent with `--preserve-var` and `--merge-etc`. The
combination should be rejected or documented as a no-op.
### `--cleanup`
`--cleanup` installs a `bootc-destructive-cleanup.service` that removes
old package-mode files on the first boot into the new system. This is
compatible with these migration flags — the cleanup runs after the new
system is booted — but operators should only enable it after validating
the migration, as it is irreversible.
### `--stateroot`
Non-default stateroot names are handled correctly; all stateroots are
enumerated when locating the new deployment directory.
---
## What is not automated
These flags deliberately do not attempt to automate everything:
- **Service restarts**: Stopping services before migration (especially
on non-reflink filesystems) is the operator's responsibility.
- **Data validation**: The operator should verify that databases and
application state are intact after the first boot.
- **Rollback removal**: The `pkgmode-rollback.conf` BLS entry persists
until the operator removes it. No automatic removal is provided.
- **Registry credentials**: `--merge-etc` will carry forward any pull
secrets already present in the running `/etc`, but registry
authentication is not otherwise managed by these flags.
Contributor guide
Research direction
Start at the `bootc install to-existing-root` entry point and trace the `install_to_filesystem()` pipeline, including the existing `etc-merge` crate. Check how deployment paths and boot entries differ between ostree and composefs. Done means the three migration flags preserve the specified state, merge `/etc`, and create a package-mode rollback entry without applying to other install commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100