oxidecomputer / oxidecomputer/omicron

Nexus should call `start_sled_agent` in its own background task

Open
#9,795 0 comments 0 reactions 1 assignee View on GitHub

@andrewjstone is already working on this.

Since Mar 26, 2026.

Add/expunge sled trust quorum
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

When a sled is added to a rack it's required that we allocate a rack subnet, create StartSledAgentRequest, and send that request to a sled-agent that acts as a proxy to start the sled-agent of the newly added sled over the bootstrap network. This eventually results in a call to start_sled_agent in the newly added sled. However, start_sled_agent itself is fallible, even though the handler in the bootstrap agent is "idempotent".

There are a couple of existing issues around start_sled_agent already:

The whole start_sled-agent operation can be subject to a few failures, and we want to ensure that we can retry in the case of transient errors. But for now, we trigger the whole procedure with a call to client.sled_add in the trust quorum background task after all sleds have acked the commit. This will only happen in one Nexus, and so we get one attempt.

We want to change this behavior so that we can retry starting a sled-agent. We propose to do this with a separate background task that looks for committed trust quorums with added sleds that do not yet have corresponding commissioned sleds in the database. This would also allow us to start a sled-agent with some sleds being non-responsive: a so called commit_partial, which happens when a new trust quorum configuration is created before all sleds have acknowledged commit. This is only allowed based on a parameter called commit_crash_tolerance which is computed based on the number of sleds in the trust quorum.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.