liquidmetal-dev / liquidmetal-dev/brigade

Auto-reschedule cattle VMs on host loss

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elixir
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Goal

Optionally recreate a VM on another host when its host dies, for stateless/cattle workloads.

## Current behaviour (M3)

On `nodedown`, a host's VMs are marked `unreachable` (capacity released, record kept) with **no auto-reschedule** — Brigade doesn't own guest disk/state and a microVM isn't trivially movable. This is the safe default.

## Proposal

Opt-in per VM via a reserved label, e.g. `brigade.reschedule=true`. When set and the host is lost:
1. Create a fresh VM from the same spec on another eligible host (new flintlock uid).
2. Only safe for cattle (reconstructable from spec, no persistent local state).

## Risks / must-handle

- **Split-brain double-run**: a partitioned-away host may still be running the VM. Reschedule must respect the quorum gate and ideally fence the old instance.
- Spec must be fully captured to recreate (it is, in the VM record — extend record to store enough of the spec, or re-fetch).
- Interaction with the reconciler's adopt-on-return: if the old host comes back with the VM, reconcile must detect the duplicate.

Deferred from the M0–M4 build.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the nodedown handling, VM record, quorum gate, and reconciler's adopt-on-return path. Determine how the full VM spec is stored or re-fetched, how fencing and duplicate detection should work, and how the reserved label enables opt-in rescheduling. Done means cattle VMs can be safely recreated on eligible hosts without split-brain duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.