oxidecomputer / oxidecomputer/omicron
Sleds that joined trust quorum but failed to initialize cannot be added again
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This scenario seems like a consequence of https://github.com/oxidecomputer/omicron/issues/4494.
Should an operator call rack_membership_add_sled, the two broad steps are:
- Add the new sled to TQ
- Send
StartSledAgentRequestto the sled
Should the sled successfully join TQ, but fail in StartSledAgentRequest before it has written its ledger, subsequent calls to rack_membership_add_sled will fail with The following sleds are already members of the trust quorum:....
The operator has no way to remove the sled from TQ, so manual intervention by support is required to resolve the situation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the rack_membership_add_sled flow and its StartSledAgentRequest handling, focusing on the trust-quorum update that occurs before ledger initialization. Reproduce the failed initialization path and verify that a later rack_membership_add_sled call can recover without manual support intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100