oxidecomputer / oxidecomputer/omicron

Nexus didn't notice second MUPdate while resolving first one

Open
#10,797 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Following up on https://github.com/oxidecomputer/omicron/issues/10796: in https://github.com/oxidecomputer/customer-support/issues/1372, we found a sequence where:

  • Due to an operational issue, TUF artifact replication was not working. This wasn't caught because nothing was using it -- the system had only been MUPdated for updates.
  • The rack was MUPdated to release N.
  • The release N TUF repo was uploaded.
  • Nexus began trying to resolve the MUPdate and its artifacts. Requests to send new sled configs to sled agents were failing because it didn't recognize many of the artifacts.
  • The rack was MUPdated to release N + 1.
  • The planner generated a new blueprint that modified sled configs only to add a new measurement file. I assume this was the measurements need for release N + 1.

At this point:

  • at least one sled agent is reporting a local MUPdate override in its inventory
  • Nexus is not generating any new blueprints

I would have expected Nexus to notice from the sled inventory that sleds had been MUPdated a second time. It should have generated new sled configs that set "remove_mupdate_override" and set all the artifacts in the sled config to "install dataset" or equivalent. It didn't.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Nexus's planner and how it consumes sled-agent inventory during successive MUpdates. Reproduce or inspect the N then N + 1 sequence described in the issue, focusing on the local MUpdate override and blueprint generation. Done means Nexus notices the second MUpdate and generates sled configs that remove the override and install the required artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.