oxidecomputer / oxidecomputer/omicron

instance-watcher should health check migration target VMMs

Open
#6,535 0 comments 0 reactions 1 assignee View on GitHub

@hawkw is already working on this.

Since Sep 6, 2024.

nexus
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Currently, the instance_watcher background task only performs health checks for an instance's active VMM. If an instance is in the midst of a migration, it may also have a migration target VMM, on another sled. Because we don't pull the state of migration target VMMs in the control plane, we instead rely on the sled-agents to push their status to Nexus, which means that if a sled-agent or VMM crashes and is restarted, we may not observe this and move it to Failed, because we're just waiting to receive a migration status update that never comes.

Therefore, we should probably extend instance_watcher to also select instances that are migration targets, and health-check them as well, so that we can notice if they've gone away. This will probably require changing the query for VMMs to also select those which are migration targets, and extend the health checking code to behave differently based on whether a VMM is a migration target or an instance's active VMM --- we won't want to increment the instance-level metrics for a migration target until it becomes active, but we do want to potentially advance the instance's state based on changes in the target's state. Eventually we will probably also want to extend the metrics schema to support recording metrics about an instance's migration target somehow, too...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.