oxidecomputer / oxidecomputer/omicron

Blueprint planner safety: Consider zones present in inventory that are not in the blueprint

Open
#8,589 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When deciding whether to upgrade zones, the planner considers whether previous upgrades have completed by confirming that the new zone config has appeared in inventory. (This can either mean a new zone has been successfully started, if the upgrade was an expunge+replace, or that the zone has successfully restarted, if the upgrade was in-place.) We should also consider zones that are present in inventory but are expunged in the blueprint.

None of this applies to upgrade-in-place zones, since there is no expungement involved.

A zone present in inventory but expunged in the parent blueprint means either the parent blueprint has not yet been executed, so the zone is still running, or it has been executed but zone shutdown failed. In the "blueprint has not been executed" case, we also won't see the new zone that's supposed to replace it, so we're okay. But in the "zone shutdown failed" case, we may be in trouble, because the old zone is still running. This could affect at least:

  • Shared resource usage (e.g., external IPs) - the planner treats the blueprint as the source of truth, so it may reassign an external IP to a new zone even if the old zone is still running. This may be fine? But I don't think we've really tested it much.
  • API versioning - we wait to upgrade Nexus until all other zone types have been upgraded to avoid API compatibility issues. If an old zone is still running, should we wait to upgrade Nexus until we can get rid of it?

We could also have expunged a zone for non-upgrade reasons. In practice we only do that for sled removal (not an issue, since the sled is gone) or disk expungement (maybe not an issue? unless the zone could still be running even if the disk has disappeared out from under it?), but in dev / support operations we can expunge zones by hand, so should still consider these too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the blueprint planner's existing inventory check for completed zone upgrades and how it distinguishes upgrade-in-place zones. Follow the inventory and blueprint zone sets, including expunged zones, and review the handling of failed shutdowns. Done means planner decisions account for still-present expunged zones without changing upgrade-in-place behavior, with coverage for the described cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.