oxidecomputer / oxidecomputer/omicron
Make RSS resistant to intermingled deployments
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
TL;DR: Two different setups of Omicron, both sharing the same network, can share prefixes via ddm and "try to induct the other" into a view of the rack.
As mentioned in https://github.com/oxidecomputer/omicron/issues/1639, it seems that two separate instantiations of RSS can try to communicate with an overlapping set of sleds, and that may not be desired.
Here's a timeline:
- Sled 1 starts RSS, is configured to boot with a rack secret threshold of "1". Sled agent comes online.
- Sled 2 starts RSS, is configured to boot with a rack secret threshold of "1".
- Sled 2 sees sled 1, advertised by ddmd. This RSS creates a plan that includes both sled 1 and 2 in the view of the "rack".
- Sled 2's RSS sends a request to "start sled agent" to Sled 1. This request fails, because sled 1 already booted the sled agent by itself. (this was a very real error, seen here - thank you @iliana for noticing!).
Some ideas for mitigating:
- As part of the bootstrap protocol, advertise whether or not the sled agent is already running. Ignore these already running sleds when configuring RSS. There's the possibility of raciness here (which machine gets picked up by which RSS?) but it makes this case more explicitly handled.
- Make RSS generate a "rack UUID" - which we're doing for Nexus, anyway - that gets transmitted during the bootstrapping phases. This way, sleds can explicitly flag the error as "I'm already part of rack
foo, but you're asking me to join rackbar. - Help a human make the right call - in a non-automated, production environment, we'll be presenting a list of sleds to an operator for confirmation. Perhaps we could figure out a way to use location information when doing this presentation - like, "you have two sleds that claim to be in the physical slot 1, what do you want to do about that?"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the related issue 1639 and the RSS bootstrap protocol described here, including how sleds are discovered through ddmd and inducted into a rack view. Compare the three proposed mitigation directions and define the expected behavior when separate RSS deployments share a network. Done means one deployment cannot mistakenly induct or start an already-running sled from another deployment, with the conflict handled explicitly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100