oxidecomputer / oxidecomputer/omicron

should unwinding instance-start sagas put the instance in `Stopped` or `Failed`?

Open
#7,727 0 comments 0 reactions 2 assignees View on GitHub

@gjcolombo is already working on this.

Since Mar 4, 2025.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When an instance-start saga unwinds, the compensating actions transition the instance back to the Stopped state. This makes sense from the perspective of "an unwinding saga node should put things back Exactly The Way They Were Before".[^1] However, it's a bit weird with regards to the user's intent: arguably, an attempt to start the instance that was unsuccessful is equivalent to successfully starting an instance and then having it fail, in terms of the desired state for that instance. The user asked us to start it, so perhaps we should continue trying to start it.[^2]

This could maybe be achieved by transitioning the instance to Failed. It's also potentially solveable by the idea @gjcolombo and I have discussed where we store a target instance state alongside the current instance state, and attempt to reconcile them when Something Happens in the Real World. This also solves stuff like #6809, which is kind of the same weird behavior in the opposite direction (we were asked to stop an instance, upon doing so we discover it's gone away, and then we immediately try to restart it, which is Goofy).

On the other hand, the question of what is the Right Thing in this situation is complicated by some of the reasons an instance-start saga may fail. In particular, it might fail due to insufficient resources being available for the instance, in which case we probably should not retry starting it immediately, but should maybe try again in a little while if sufficient resources become available? Or perhaps not --- maybe retrying starting an instance that we couldn't find resources for should be an explicit user action? I feel a bit conflicted about that.

[^1]: (more or less, modulo generation numbers &c)
[^2]: At least, if auto-restart is enabled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.