randomparity / randomparity/kdive
Add BYO teardown: baseline restore, OOB power-cycle, and reconciler drift repair
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
Part of #1814
## Problem
When a VM allocation ends, the domain is destroyed and the next allocation starts clean. A physical
host has no such reset: after a Run installs an agent's kernel and force-crashes it, the machine
stays exactly as the crash left it — booting an arbitrary, possibly broken kernel, with whatever
state the crash produced.
Releasing that host without restoring it means the next allocation silently adopts a poisoned
machine. Since KDIVE deliberately cannot re-image a BYO host, the only honest options are to
restore it to a declared baseline or to refuse to hand it out again.
A dead worker mid-teardown produces the same hazard, which is why the reconciler needs a path here
too.
## Evidence
- `src/kdive/domain/catalog/resources.py:34` — `cordoned: bool`, the existing mechanism for taking
a Resource out of rotation.
- ADR-0021 — the reconciler's drift-repair remit: tears down orphaned Systems, fails Runs on
torn-down Systems, reclaims expired leases.
- `src/kdive/providers/core/runtime.py:76` — snapshots are unavailable to a bare-metal provider, so
"restore" cannot mean reverting a disk image; it means re-pointing the bootloader.
- The `baseline_kernel` field declared in #1817; how it is identified is decided in #1815 (open
question 1) — teardown cannot restore what it cannot name.
- Related prior work under epic #1760: #1784 (recover stale bare-metal serial and KGDB sessions)
overlaps the stale-session half. See #1814's overlap table.
## Expected outcome
- Teardown re-points the bootloader at the declared `baseline_kernel`, power-cycles via the OOB
driver, re-runs the #1823 adopt preconditions, and only then frees the Resource.
- **Any failure in that sequence cordons the Resource with a readable reason** rather than
re-offering it — this is #1814 success criterion 4, and the deliberate alternative to an
unbounded retry loop.
- Debug sessions and console consumers are detached before the power-cycle; a KGDB session must not
survive into the next allocation.
- A reconciler path for a host left mid-teardown by a dead worker, consistent with ADR-0021 and the
existing worker-fence evidence rules.
- Teardown is idempotent and serialized per-System under the existing advisory locks; a repeated
teardown must not double-power-cycle a host another allocation has already claimed.
Blocked by #1827
Contributor guide
Research direction
Start with src/kdive/domain/catalog/resources.py:34 and src/kdive/providers/core/runtime.py:76, then read ADR-0021 and the related issues #1815, #1817, #1823, and #1827. Trace the existing teardown, advisory-lock, worker-fence, and reconciler entry points before designing the change. Done means baseline restore, OOB power-cycle, adopt-precondition checks, safe detachment, readable cordoning on failure, and idempotent recovery are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100