oxidecomputer / oxidecomputer/omicron

can RSS recover from restarts during rack initialization?

Open
#4,553 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

There's a lot of work in RSS to make it so that if it restarts while it's running, then it can pick up where it left off and keep finishing rack setup. I think this doesn't work though because bootstrap agent will refuse to run RSS if this sled agent has a ledgered StartSledAgentRequest. More specifically:

When bootstrap agent starts up, it creates a RssAccess::new which either winds up in state Initialized (if a StartSledAgentRequest was found in the local ledger) or Uninitialized (otherwise):
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/server.rs#L183-L194
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L85-L89

Then later if we try to start RSS, we check this state:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L178-L180

and bail out if it's Initialized:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/bootstrap/rack_ops.rs#L184-L186

I think the StartSledAgentRequest gets written during RSS here:
https://github.com/oxidecomputer/omicron/blob/36f6abe06548eae0f954f1b968f98def846f7d93/sled-agent/src/rack_setup/service.rs#L918-L929

I think the upshot is that if we make it that far into RSS and then restart, we can't resume. We'll get an "AlreadyInitialized" error even though we haven't finished initialization.

I think we've discussed this elsewhere: it's probably not important that we are able to resume RSS. We could instead factory-reset and just run it again. That currently has issues too though (#3820, #3818).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent/src/bootstrap/server.rs and sled-agent/src/bootstrap/rack_ops.rs to trace how RssAccess is initialized and why an existing StartSledAgentRequest causes an AlreadyInitialized error. Then inspect sled-agent/src/rack_setup/service.rs around the request write. Determine whether interrupted RSS setup should resume or be rerun after reset, and document the resulting behavior and scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.