oxidecomputer / oxidecomputer/omicron
Nexus should call `start_sled_agent` in its own background task
@andrewjstone is already working on this.
Since Mar 26, 2026.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
When a sled is added to a rack it's required that we allocate a rack subnet, create StartSledAgentRequest, and send that request to a sled-agent that acts as a proxy to start the sled-agent of the newly added sled over the bootstrap network. This eventually results in a call to start_sled_agent in the newly added sled. However, start_sled_agent itself is fallible, even though the handler in the bootstrap agent is "idempotent".
There are a couple of existing issues around start_sled_agent already:
- https://github.com/oxidecomputer/omicron/issues/4494
- https://github.com/oxidecomputer/omicron/issues/5132
The whole start_sled-agent operation can be subject to a few failures, and we want to ensure that we can retry in the case of transient errors. But for now, we trigger the whole procedure with a call to client.sled_add in the trust quorum background task after all sleds have acked the commit. This will only happen in one Nexus, and so we get one attempt.
We want to change this behavior so that we can retry starting a sled-agent. We propose to do this with a separate background task that looks for committed trust quorums with added sleds that do not yet have corresponding commissioned sleds in the database. This would also allow us to start a sled-agent with some sleds being non-responsive: a so called commit_partial, which happens when a new trust quorum configuration is created before all sleds have acknowledged commit. This is only allowed based on a parameter called commit_crash_tolerance which is computed based on the number of sleds in the trust quorum.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.