bootc-dev / bootc-dev/bootc

install: Add support for preserving system state during install to-existing-root.

Open
#2,317 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.3k
Forks
230
Avg merge
3d 12h
Merged PRs (30d)
38

Description

The following is an AI assisted spec that describes high level CLI changes to add support for converting a "package mode" system into an "image mode" (bootc) system while preserving system state and configuration.

---

# Design: Package-mode to Image-mode Migration via `bootc install to-existing-root`

This document describes the new migration flags added to
`bootc install to-existing-root` to support converting a running
package-mode (RPM/DEB) system to a bootc image-mode deployment in place,
without a reinstall.

---

## Overview

`bootc install to-existing-root` runs the bootc install pipeline against
the host's root filesystem, replacing the running OS with a new bootc
deployment on the next reboot. By default this is a clean install: the
new deployment's `/var` is empty and `/etc` comes entirely from the image.

Three new flags extend this command for the migration use case:

| Flag | Purpose |
|---|---|
| `--preserve-var` | Copy `/var` data into the new deployment |
| `--merge-etc` | Carry forward `/etc` customisations via 3-way merge |
| `--pkgmode-rollback` | Write a boot entry to roll back to the old OS |

These flags are independent and can be combined freely. The typical
migration invocation is:

```bash
podman run --rm --privileged --pid=host \
-v /:/target \
-v /var/lib/containers:/var/lib/containers \
--security-opt label=type:unconfined_t \
\
bootc install to-existing-root \
--preserve-var \
--merge-etc \
--pkgmode-rollback \
--acknowledge-destructive
```

---

## Flag: `--preserve-var`

After a plain `bootc install to-existing-root`, the new deployment's
`/var` is initially empty — ostree bind-mounts it from a fresh directory
under the stateroot. The package-mode `/var` (databases, application
data, home directories, etc.) is stranded and inaccessible to the new
system.

`--preserve-var` copies `/var` into the new deployment's stateroot
`var/` directory after the core install completes, using one of two
strategies:

- **Reflink copy** (btrfs / XFS): `cp --reflink=always` performs an
instantaneous copy-on-write clone. No extra disk space is consumed
until data diverges. This is probed at runtime.

- **Plain copy** (ext4 and other non-reflink filesystems): `cp -a`
copies each subdirectory of `/var` into the new deployment's stateroot.
Correct but slow for large `/var` trees.

### Rejected alternative: `var.mount` bind-mount unit

An earlier approach injected a `var.mount` systemd unit into the new
deployment's `/etc/systemd/system/` that would bind-mount the old
`/sysroot/var` onto `/var` at boot, avoiding the copy entirely. This was
rejected because both the ostree and composefs backends already
bind-mount the stateroot `var/` onto `/var` in the initramfs, before
systemd starts. A `var.mount` unit fires too late — the mount point is
already in use — and is either silently ignored or fails. The correct
approach is to populate the stateroot `var/` directly before the first
boot, which is what the copy strategies above do.

In both strategies, certain subdirectories are always skipped:

| Skipped path | Reason |
|---|---|
| `var/tmp`, `var/cache` | Ephemeral, regenerated automatically on first boot |
| `var/lib/containers` | Overlay mounts cannot be reflinked or copied across device boundaries |

Additional directories can be excluded via a `--preserve-var-exclude=`
flag. This allows operators to skip large or sensitive directories on a
case-by-case basis — for example, `--preserve-var-exclude=var/log/journal`
on systems with large journals, or `--preserve-var-exclude=var/lib/pgsql`
when the database will be migrated separately.

### Known limitation — live databases on non-reflink filesystems

The plain copy is **unsafe for applications with open write handles**
(PostgreSQL, MySQL, SQLite). Copying a live database produces a corrupted
copy. On ext4 systems, operators must stop stateful services before
running `--preserve-var`, or use a filesystem that supports reflinks
(btrfs, XFS) to take advantage of the atomic copy path. High-risk
directories can be skipped via `--preserve-var-exclude`.

---

## Flag: `--merge-etc`

A plain install populates the new deployment's `/etc` from the image.
The running system's admin customisations — NIC profiles, SSH host
keys, custom CA certificates, application secrets, sysctl drop-ins, etc.
— are not applied to the new deployment.

`--merge-etc` runs a 3-way merge using bootc's existing `etc-merge`
crate after the core install completes. This is the same algorithm
bootc uses during `bootc upgrade`, applied at install time rather than
only at upgrade time:

| Input | Source | Role |
|---|---|---|
| A — pristine baseline | `/usr/etc` | Image's shipped defaults |
| B — current live | `/etc` | Running admin customisations |
| C — new deployment | `/etc` | Merge target |

The diff A→B captures every change the admin made relative to the image
defaults and applies those changes onto C.

Machine-specific files (SSH host keys, machine-id, NIC profiles) are
intentionally included — they are precisely what must be transferred to
make the migrated system functional on first boot.

### Interaction with ostree's /etc semantics

`bootc upgrade` performs this same 3-way merge at upgrade time.
`--merge-etc` at migration time seeds that mechanism correctly, so
subsequent upgrades inherit the right baseline.

---

## Flag: `--pkgmode-rollback`

When set, a "Previous OS" boot entry is written after the install so the
operator can boot back to the old package-mode system if the migration
fails or an application breaks.

The entry is a standard BLS snippet written into the active
`boot/loader.N/entries/` directory alongside the ostree BLS entries:

```
title Previous OS — package-mode rollback (kernel )
sort-key zz-pkgmode
linux /boot/pkgmode-rollback/vmlinuz
initrd /boot/pkgmode-rollback/initramfs.img
options
```

The `zz-pkgmode` sort-key places the entry last in the boot menu.
The `pkgmode-rollback.conf` filename is invisible to ostree (which only
reads `ostree-*.conf` entries) and to bootupd (which does not manage
`loader/entries/`), so it will not be removed by subsequent
`bootc upgrade` operations until explicitly deleted by the operator.

The kernel and initramfs are stashed to `var/lib/pkgmode-rollback/`
before the install wipes `/boot`, then installed to
`boot/pkgmode-rollback/` after the install recreates `/boot`. The
stash lives under `/var` so it survives the `/boot` wipe.

---

## Execution order

The migration steps interlock tightly with the core install pipeline:

```
1. --pkgmode-rollback / --preserve-var:
save_pkgmode_kernel() ← BEFORE install wipes /boot

2. install_to_filesystem() ← core ostree deploy + bootupd

3. --preserve-var:
preserve_var() ← copy /var into new stateroot

4. --pkgmode-rollback:
write_pkgmode_rollback_entry() ← write BLS entry + install kernel

5. --merge-etc:
merge_etc_into_deployment() ← 3-way /etc merge
```

Step 1 must run before step 2. Steps 3–5 are independent of each other
and run after the core install completes.

---

## Interaction with other install options and backends

### composefs backend (`--composefs-backend`)

The composefs backend uses a different on-disk layout from the ostree backend,
and `--preserve-var` and `--merge-etc` must handle both.

**Detection**: after the install completes, the active backend is identified
by which directories are present under ``:

- ostree backend: `/ostree/deploy//deploy//`
- composefs backend: `/composefs/` and `/state/deploy//`

**`--preserve-var`**: the ostree backend gives each stateroot its own
`var/` directory at `/ostree/deploy//var/`. The composefs
backend instead uses a single shared directory at `/state/os/default/var`,
with each deployment's `var` entry being a symlink to it. For the composefs
case, `--preserve-var` must populate `/state/os/default/var` directly
rather than a per-deployment path.

**`--merge-etc`**: the ostree backend stores the deployment's mutable `etc/`
under `/ostree/deploy//deploy//etc/`. In the composefs
backend it is at `/state/deploy//etc/`. The most recently
created entry under `/state/deploy/` identifies the new deployment, the
same heuristic used for the ostree backend.

The pristine baseline (`usr/etc`) is the same in both cases: it is always
read from the deployment's read-only root image, which in the composefs backend
is an EROFS image mounted via composefs. The `/usr/etc` path within that image
is accessible via the deployment's EROFS mount at install time.

The composefs "sealed" variant (UKI + fsverity) embeds a SHA-512 digest
of the root filesystem in the kernel command line. The rollback BLS entry
uses the original package-mode kernel and is unaffected by this; the
stash/restore of vmlinuz/initramfs for `--pkgmode-rollback` is
backend-agnostic.

### Bootloader variants

`--pkgmode-rollback` writes the BLS entry into the same
`boot/loader.N/entries/` directory used by bootupd, and is therefore
compatible with both GRUB and systemd-boot.

When `--bootloader=none` is used, the entry file is still written but
whether it appears in the boot menu depends on the external bootloader
configuration.

### `to-disk` and `to-filesystem`

These three flags are only available on `to-existing-root`. They are
not applicable to `to-disk` or `to-filesystem`, which install onto a
blank or separately mounted target with no running system state to
preserve.

### `--replace=wipe`

`--replace=wipe` wipes the existing root before the install, which is
incompatible in intent with `--preserve-var` and `--merge-etc`. The
combination should be rejected or documented as a no-op.

### `--cleanup`

`--cleanup` installs a `bootc-destructive-cleanup.service` that removes
old package-mode files on the first boot into the new system. This is
compatible with these migration flags — the cleanup runs after the new
system is booted — but operators should only enable it after validating
the migration, as it is irreversible.

### `--stateroot`

Non-default stateroot names are handled correctly; all stateroots are
enumerated when locating the new deployment directory.

---

## What is not automated

These flags deliberately do not attempt to automate everything:

- **Service restarts**: Stopping services before migration (especially
on non-reflink filesystems) is the operator's responsibility.
- **Data validation**: The operator should verify that databases and
application state are intact after the first boot.
- **Rollback removal**: The `pkgmode-rollback.conf` BLS entry persists
until the operator removes it. No automatic removal is provided.
- **Registry credentials**: `--merge-etc` will carry forward any pull
secrets already present in the running `/etc`, but registry
authentication is not otherwise managed by these flags.

Contributor guide

Open the contributing guide

Research direction

Start at the `bootc install to-existing-root` entry point and trace the `install_to_filesystem()` pipeline, including the existing `etc-merge` crate. Check how deployment paths and boot entries differ between ostree and composefs. Done means the three migration flags preserve the specified state, merge `/etc`, and create a package-mode rollback entry without applying to other install commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.