randomparity / randomparity/kdive

Add BYO teardown: baseline restore, OOB power-cycle, and reconciler drift repair

Open
#1,830 0 comments 0 reactions 0 assignees View on GitHub
area:allocation area:providers effort:L priority:P1 provider:baremetal-bmc provider:powervm status:blocked type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

Part of #1814

## Problem

When a VM allocation ends, the domain is destroyed and the next allocation starts clean. A physical
host has no such reset: after a Run installs an agent's kernel and force-crashes it, the machine
stays exactly as the crash left it — booting an arbitrary, possibly broken kernel, with whatever
state the crash produced.

Releasing that host without restoring it means the next allocation silently adopts a poisoned
machine. Since KDIVE deliberately cannot re-image a BYO host, the only honest options are to
restore it to a declared baseline or to refuse to hand it out again.

A dead worker mid-teardown produces the same hazard, which is why the reconciler needs a path here
too.

## Evidence

- `src/kdive/domain/catalog/resources.py:34` — `cordoned: bool`, the existing mechanism for taking
a Resource out of rotation.
- ADR-0021 — the reconciler's drift-repair remit: tears down orphaned Systems, fails Runs on
torn-down Systems, reclaims expired leases.
- `src/kdive/providers/core/runtime.py:76` — snapshots are unavailable to a bare-metal provider, so
"restore" cannot mean reverting a disk image; it means re-pointing the bootloader.
- The `baseline_kernel` field declared in #1817; how it is identified is decided in #1815 (open
question 1) — teardown cannot restore what it cannot name.
- Related prior work under epic #1760: #1784 (recover stale bare-metal serial and KGDB sessions)
overlaps the stale-session half. See #1814's overlap table.

## Expected outcome

- Teardown re-points the bootloader at the declared `baseline_kernel`, power-cycles via the OOB
driver, re-runs the #1823 adopt preconditions, and only then frees the Resource.
- **Any failure in that sequence cordons the Resource with a readable reason** rather than
re-offering it — this is #1814 success criterion 4, and the deliberate alternative to an
unbounded retry loop.
- Debug sessions and console consumers are detached before the power-cycle; a KGDB session must not
survive into the next allocation.
- A reconciler path for a host left mid-teardown by a dead worker, consistent with ADR-0021 and the
existing worker-fence evidence rules.
- Teardown is idempotent and serialized per-System under the existing advisory locks; a repeated
teardown must not double-power-cycle a host another allocation has already claimed.

Blocked by #1827

Contributor guide

Open the contributing guide

Research direction

Start with src/kdive/domain/catalog/resources.py:34 and src/kdive/providers/core/runtime.py:76, then read ADR-0021 and the related issues #1815, #1817, #1823, and #1827. Trace the existing teardown, advisory-lock, worker-fence, and reconciler entry points before designing the change. Done means baseline restore, OOB power-cycle, adopt-precondition checks, safe detachment, readable cordoning on failure, and idempotent recovery are covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.