oxidecomputer / oxidecomputer/omicron

`SagaUnwound` VMMs don't reincarnate as expected (#6669 is busted)

Open
#9,174 3 comments 0 reactions 1 assignee View on GitHub

@hawkw is already working on this.

Since Oct 8, 2025.

bug
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Commit #6669 was intended to allow instances whose instance_start sagas have unwound to automatically restart. However, it turns out that it doesn't work (at least sometimes, probably due to a race).

The reason this doesn't work is because the check for whether an instance needs to be restarted currently checks that the instance is either in the Failed state, or has an active VMM that is in the SagaUnwound state:

https://github.com/oxidecomputer/omicron/blob/6bae7930841f4eb46e8665c03c9b69002d5fd5a9/nexus/db-model/src/instance.rs#L321-L342

However, the abandoned_vmm_reaper background task will handle instances which have an active VMM that is in the SagaUnwound state by...unlinking the active VMM and moving the instance to NoVmm (Stopped):
https://github.com/oxidecomputer/omicron/blob/a3ee7427a16c525636e35e06ddf13756096cbd76/nexus/src/app/background/tasks/abandoned_vmm_reaper.rs#L67-L75
https://github.com/oxidecomputer/omicron/blob/a3ee7427a16c525636e35e06ddf13756096cbd76/nexus/db-queries/src/db/datastore/vmm.rs#L392-L402

So, there's an unfortunate race here where if the abandoned_vmm_reaper activates first, the instance may be moved to the Stopped state before it has a chance to reincarnate. This was not caught by the integration tests for reincarnating these instances that I added in #6669, because, again, it's a race. The tests explicitly activate the reincarnation task, so it runs before the reaper task. But this seems to not be the case in real life, at least sometimes.

We could probably fix this by having the reaper move instances where the active VMM is SagaUnwound to Failed rather than to Stopped, so that the instance remains eligible for reincarnation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.