oxidecomputer / oxidecomputer/omicron
instance-watcher should health check migration target VMMs
@hawkw is already working on this.
Since Sep 6, 2024.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Currently, the instance_watcher background task only performs health checks for an instance's active VMM. If an instance is in the midst of a migration, it may also have a migration target VMM, on another sled. Because we don't pull the state of migration target VMMs in the control plane, we instead rely on the sled-agents to push their status to Nexus, which means that if a sled-agent or VMM crashes and is restarted, we may not observe this and move it to Failed, because we're just waiting to receive a migration status update that never comes.
Therefore, we should probably extend instance_watcher to also select instances that are migration targets, and health-check them as well, so that we can notice if they've gone away. This will probably require changing the query for VMMs to also select those which are migration targets, and extend the health checking code to behave differently based on whether a VMM is a migration target or an instance's active VMM --- we won't want to increment the instance-level metrics for a migration target until it becomes active, but we do want to potentially advance the instance's state based on changes in the target's state. Eventually we will probably also want to extend the metrics schema to support recording metrics about an instance's migration target somehow, too...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.