liquidmetal-dev / liquidmetal-dev/flintlock

VM Supervisor

Open
#198 11 comments 1 reaction 0 assignees View on GitHub
adr-required area/firecracker area/reconciliation Epic kind/feature lifecycle/stale
Dominant language
Go
Stars
1.5k
Forks
73
Avg merge
17h 24m
Merged PRs (30d)
30

Description

The Supervisor will be responsible for monitoring running MicroVMs and react to changes that drift from the desired state.

**Why do we need this?**

We don't currently continuously monitor the state of running VMs. If a VM drifts away from its desired state - e.g. A VM crashes and is in a failed state - we need to wait until the next time the reconciler runs for the VM to be recreated/restarted.

Additionally, we don't currently track if a VM is continuously failing. Flintlock will continue to recreate the VM every time a resync occurs. The reconciler doesn't know if a VM has already been started, as far as it is concerned it only cares about reconciling the existing state to the desired state.

**What do we need**

- The VM Supervisor should exist as a background process/goroutine.
- As firecracker does not have a background daemon or event system, the supervisor will need to have knowledge of the VMs that should/do exist and their desired state, and should continuously (on a short timer) check the status of each VM.
- The supervisor should probably utilise the containerd state to store events (e.g. VM stopped, VM started, VM restarted X times, etc), and the event bus to notify the reconciler to take action outside the delayed reconciler resync loop.

How this looks on an implementation level is unknown, and it's likely one or more ADRs will need to be produced as a result.

**Subtasks**

- [ ] Create supervisor that starts when `flintlockd` is started, detect and continuously list all VMs and their state directly from Firecracker.
- [ ] Extend MicroVM model to track microvm events - started, stopped, reboots, etc.
- [ ] Detect when VMs are in a failed state and trigger an event on the event bus to reconcile the VM state
- [ ] Track number of failures/restarts and emit metrics.

Contributor guide

Open the contributing guide

Research direction

Start by reading how flintlockd starts, how MicroVM state is represented, and how the reconciler and event bus interact. Review the Firecracker state checks and the proposed ADRs before designing the supervisor. Done means continuous VM monitoring, failure events that trigger reconciliation, tracked lifecycle events and restart counts, and emitted metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.