oxidecomputer / oxidecomputer/omicron
tq_load_latest_possible_committed_config is probably wrong
@andrewjstone is already working on this.
Since Aug 11, 2026.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
@jgallagher and I were trying to reproduce a customer issue on a racklette today. John issued a trust quorum sled addition which would go from 3-4 sleds in epoch 2. However, before doing so, we intentionally took down one of the initial 3 sleds. Nexus chooses a random sled-agent to act as coordinator for the reconfiguration, and it chose the sled that happened to be down.
Before issuing a reconfiguration to the coordinator, Nexus records the configuration in CRDB. This means that epoch 2 is already recorded. We looked at the database state and saw that none of the peers had acked the prepare message because the coordinator was down and didn't send prepares. This should be fine. The trust quorum protocol allows skipping a configuration in case of a failed coordinator, as long as Nexus hasn't already moved the state to committing.
We tried to re-issue the same add-sled request hoping to hit a different coordinator. However, the request was rejected as it said the new sled was already a member. But the trust quorum should only treat sleds as members if the configuration has committed (moved to committing state in CRDB). Looking at the code, I realized that we sometimes treat the latest configuration as committed even when it isn't. This is not correct, although the comment doesn't indicate why exactly we do this and I can't remember. I do remember that I did it on purpose, but possibly just as an optimization which the comment hints at.
The code that loads the latest "possible" committed config is https://github.com/oxidecomputer/omicron/blob/d0918eb077348fc4b6452fa7ba97385ac52262f3/nexus/src/app/trust_quorum.rs#L441. It appears incorrect. What should happen is that we return the actual last committed config, which would be epoch 1 in this case. Then we would be able to build a new configuration identical to epoch 2, but with the value of epoch 3 and attempt a reconfiguration that would hopefully hit a different coordinator. That should succeed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.