oxidecomputer / oxidecomputer/omicron
can RSS recover from restarts during rack initialization?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
There's a lot of work in RSS to make it so that if it restarts while it's running, then it can pick up where it left off and keep finishing rack setup. I think this doesn't work though because bootstrap agent will refuse to run RSS if this sled agent has a ledgered StartSledAgentRequest. More specifically:
When bootstrap agent starts up, it creates a RssAccess::new which either winds up in state Initialized (if a StartSledAgentRequest was found in the local ledger) or Uninitialized (otherwise):
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/server.rs#L183-L194
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L85-L89
Then later if we try to start RSS, we check this state:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L178-L180
and bail out if it's Initialized:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L184-L186
I think the StartSledAgentRequest gets written during RSS here:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/rack_setup/service.rs#L918-L929
I think the upshot is that if we make it that far into RSS and then restart, we can't resume. We'll get an "AlreadyInitialized" error even though we haven't finished initialization.
I think we've discussed this elsewhere: it's probably not important that we are able to resume RSS. We could instead factory-reset and just run it again. That currently has issues too though (#3820, #3818).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sled-agent/src/bootstrap/server.rs and sled-agent/src/bootstrap/rack_ops.rs to trace how RssAccess is initialized and why an existing StartSledAgentRequest causes an AlreadyInitialized error. Then inspect sled-agent/src/rack_setup/service.rs around the request write. Determine whether interrupted RSS setup should resume or be rerun after reset, and document the resulting behavior and scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100