randomparity / randomparity/kdive

Kernel upgrades undo the /boot relabel, so guest-image builds break silently after an unattended upgrade

Open
#2,544 0 comments 0 reactions 0 assignees View on GitHub
area:provisioning effort:S priority:P2 provider:local-libvirt type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

`deploy/ansible/roles/local_worker_host/tasks/boot_kernels.yml` relabels the kernels present in `/boot` at the time it runs to `0640 root:kvm`, so libguestfs' supermin appliance can copy a host kernel as the invoking worker account (ADR-0222, #2479).

The relabel is a point-in-time `find` plus `file` loop over its results. A kernel package upgrade installs a **new** file under a **new** name at the distro default `0600 root:root`, which the earlier run never saw. The role says so itself:

```yaml
# ... A kernel UPGRADE
# installs a fresh 0600 file under a new name: re-run the recipe afterwards.
```

(`boot_kernels.yml:64-65`)

"Re-run the recipe afterwards" is a manual step with no prompt, no check, and nothing that fails until a build does.

## Observed

On a self-hosted KVM runner (Ubuntu 26.04), unattended-upgrades installed a newer kernel and the host rebooted into it:

```
-rw-r----- 1 root kvm /boot/vmlinuz-7.0.0-30-generic <- relabelled by the role
-rw------- 1 root root /boot/vmlinuz-7.0.0-31-generic <- installed later, never seen
$ uname -r
7.0.0-31-generic
```

The running kernel was the unreadable one. `scripts/operations/check-local-libvirt.sh` correctly reported it:

```
FAIL a host kernel under /boot (vmlinuz-* on x86_64, vmlinux-* on ppc64le) is not readable
by this user (libguestfs build-fs appliance, ADR-0222)
```

## The preflight detects it and the run continues anyway

The scheduled native `live_vm` job ran that preflight, got the FAIL, printed

```
=== local-libvirt host is NOT ready (see FAIL entries above) ===
WARN: local-libvirt preflight reported problems; funding the project anyway.
```

and proceeded. Eight seconds later the real failure arrived with the cause stripped out:

```
systems.get error: infrastructure_failure — libguestfs failed extracting the baseline kernel from the rootfs base
##[error]Process completed with exit code 1.
```

A reader of that job log sees `infrastructure_failure` at the bottom and an accurate diagnosis 30 lines above it, disconnected. The preflight did its job; the spine discarded the answer.

## Two separable defects

**1. The relabel is not durable across kernel upgrades.** Debian and Ubuntu run every executable in `/etc/kernel/postinst.d/` after installing a kernel, which is the platform's own mechanism for exactly this and is already populated on these hosts (`dracut`, `kdump-tools`, `unattended-upgrades`, `zz-update-grub`). A hook installed there by the role would relabel each new kernel at install time, making the guarantee survive the upgrade that currently breaks it.

This is the same defect shape as the missing tmpfiles rule for `/run/kdive/live-libvirt`: install-time state that a later platform action removes, with a native mechanism available to re-establish it.

The hook must apply the role's mode exactly — `0640 root:kvm`, not `0644` — and the role comment's reasoning for why Debian-family only still governs: Fedora ships these world-readable, so relabelling there would narrow them.

**2. A preflight FAIL should not be a warning on the native tier.** The `WARN ... funding the project anyway` path converts a specific, actionable diagnosis into a generic `infrastructure_failure` one step later. Whether that path should hard-fail, or fail only for FAIL entries the following steps depend on, is a design question worth its own decision — the current behavior costs a reader the diagnosis the preflight already produced.

These are sized as one PR each and are independent; the second does not require the first.

## Immediate mitigation

Relabelling the running kernel to `0640 root:kvm` by hand restored the preflight to `=== local-libvirt host is ready ===` (exit 0) and made the kernel readable by the runner account. That is the manual step the role comment describes, and it will be undone by the next kernel upgrade.

## Provenance

Found while diagnosing scheduled native `live_vm` job failures during campaign work on the live tiers. Two other faults on the same host — a missing `/run/kdive/live-libvirt` runtime root and a stale worker-lifecycle protocol identity — were fixed separately; this was the third and is the one that survives into the repository as a defect rather than a host state.

Contributor guide

Open the contributing guide

Research direction

Split the work into the two independent PRs described. For the relabel defect, start with deploy/ansible/roles/local_worker_host/tasks/boot_kernels.yml and the Debian-family /etc/kernel/postinst.d mechanism; run scripts/operations/check-local-libvirt.sh after a kernel upgrade to verify the guarantee. For the preflight defect, trace the native live_vm path from that script and define whether dependent FAIL entries must stop the run while preserving the diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, linux, ubuntu
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.