liquidmetal-dev / liquidmetal-dev/flintlock
VM Supervisor
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 73
- Avg merge
- 17h 24m
- Merged PRs (30d)
- 30
Description
The Supervisor will be responsible for monitoring running MicroVMs and react to changes that drift from the desired state.
**Why do we need this?**
We don't currently continuously monitor the state of running VMs. If a VM drifts away from its desired state - e.g. A VM crashes and is in a failed state - we need to wait until the next time the reconciler runs for the VM to be recreated/restarted.
Additionally, we don't currently track if a VM is continuously failing. Flintlock will continue to recreate the VM every time a resync occurs. The reconciler doesn't know if a VM has already been started, as far as it is concerned it only cares about reconciling the existing state to the desired state.
**What do we need**
- The VM Supervisor should exist as a background process/goroutine.
- As firecracker does not have a background daemon or event system, the supervisor will need to have knowledge of the VMs that should/do exist and their desired state, and should continuously (on a short timer) check the status of each VM.
- The supervisor should probably utilise the containerd state to store events (e.g. VM stopped, VM started, VM restarted X times, etc), and the event bus to notify the reconciler to take action outside the delayed reconciler resync loop.
How this looks on an implementation level is unknown, and it's likely one or more ADRs will need to be produced as a result.
**Subtasks**
- [ ] Create supervisor that starts when `flintlockd` is started, detect and continuously list all VMs and their state directly from Firecracker.
- [ ] Extend MicroVM model to track microvm events - started, stopped, reboots, etc.
- [ ] Detect when VMs are in a failed state and trigger an event on the event bus to reconcile the VM state
- [ ] Track number of failures/restarts and emit metrics.
Contributor guide
Research direction
Start by reading how flintlockd starts, how MicroVM state is represented, and how the reconciler and event bus interact. Review the Firecracker state checks and the proposed ADRs before designing the supervisor. Done means continuous VM monitoring, failure events that trigger reconciliation, tracked lifecycle events and restart counts, and emitted metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100