oxidecomputer / oxidecomputer/omicron
Nexus didn't notice second MUPdate while resolving first one
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Following up on https://github.com/oxidecomputer/omicron/issues/10796: in https://github.com/oxidecomputer/customer-support/issues/1372, we found a sequence where:
- Due to an operational issue, TUF artifact replication was not working. This wasn't caught because nothing was using it -- the system had only been MUPdated for updates.
- The rack was MUPdated to release N.
- The release N TUF repo was uploaded.
- Nexus began trying to resolve the MUPdate and its artifacts. Requests to send new sled configs to sled agents were failing because it didn't recognize many of the artifacts.
- The rack was MUPdated to release N + 1.
- The planner generated a new blueprint that modified sled configs only to add a new measurement file. I assume this was the measurements need for release N + 1.
At this point:
- at least one sled agent is reporting a local MUPdate override in its inventory
- Nexus is not generating any new blueprints
I would have expected Nexus to notice from the sled inventory that sleds had been MUPdated a second time. It should have generated new sled configs that set "remove_mupdate_override" and set all the artifacts in the sled config to "install dataset" or equivalent. It didn't.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Nexus's planner and how it consumes sled-agent inventory during successive MUpdates. Reproduce or inspect the N then N + 1 sequence described in the issue, focusing on the local MUpdate override and blueprint generation. Done means Nexus notices the second MUpdate and generates sled configs that remove the override and install the required artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100