liquidmetal-dev / liquidmetal-dev/brigade
Auto-reschedule cattle VMs on host loss
- Dominant language
- Elixir
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Goal
Optionally recreate a VM on another host when its host dies, for stateless/cattle workloads.
## Current behaviour (M3)
On `nodedown`, a host's VMs are marked `unreachable` (capacity released, record kept) with **no auto-reschedule** — Brigade doesn't own guest disk/state and a microVM isn't trivially movable. This is the safe default.
## Proposal
Opt-in per VM via a reserved label, e.g. `brigade.reschedule=true`. When set and the host is lost:
1. Create a fresh VM from the same spec on another eligible host (new flintlock uid).
2. Only safe for cattle (reconstructable from spec, no persistent local state).
## Risks / must-handle
- **Split-brain double-run**: a partitioned-away host may still be running the VM. Reschedule must respect the quorum gate and ideally fence the old instance.
- Spec must be fully captured to recreate (it is, in the VM record — extend record to store enough of the spec, or re-fetch).
- Interaction with the reconciler's adopt-on-return: if the old host comes back with the VM, reconcile must detect the duplicate.
Deferred from the M0–M4 build.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the nodedown handling, VM record, quorum gate, and reconciler's adopt-on-return path. Determine how the full VM spec is stored or re-fetched, how fencing and duplicate detection should work, and how the reserved label enables opt-in rescheduling. Done means cattle VMs can be safely recreated on eligible hosts without split-brain duplicates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100