oxidecomputer / oxidecomputer/omicron

prevent Nexus from performing schema updates when it's been MUPdated

Open
#8,954 4 comments 0 reactions 1 assignee View on GitHub

@sunshowers is already working on this.

Since May 5, 2026.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

After #8912, when Nexus starts up and determines that it's part of the set of Nexus instances currently in control, it will perform any needed database schema update. In the normal self-service update case, this is necessary and correct. However, if we had a normally running system after 8912 and someone were to MUPdate a sled that has a Nexus on it to a new version of Nexus with a new database schema, it would find that it's still in the active set and do a schema update that would pull the rug out from the other Nexus instances (violating our runtime constraint that all Nexus instances only ever use the database running the schema version that they know about).

The assumption is that this would probably have been an operator / support mistake. It's always going to be dangerous to MUPdate a sled that's part of the control plane while the control plane is running because of problems like this (and analogous problems with Crucible or Oximeter, which have their own on-disk formats and schema versions), and you could be invalidating inter-API version dependencies, etc. But this Nexus case is particularly bad since in principle it could lead to permanent control plane database corruption. If it's easy to prevent, that seems worthwhile. On today's watercooler call we discussed preventing this by having the db_metadata_nexus records contain the Nexus image id, and having Nexus know its own image id, and then having it stop on startup if it finds its image doesn't match the one in its record. (We'd presumably also want it to be able to expose this status somewhere so that an operator or support can see what's happened.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.