oxidecomputer / oxidecomputer/omicron

Make RSS resistant to intermingled deployments

Open
#1,640 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

TL;DR: Two different setups of Omicron, both sharing the same network, can share prefixes via ddm and "try to induct the other" into a view of the rack.

As mentioned in https://github.com/oxidecomputer/omicron/issues/1639, it seems that two separate instantiations of RSS can try to communicate with an overlapping set of sleds, and that may not be desired.

Here's a timeline:

  • Sled 1 starts RSS, is configured to boot with a rack secret threshold of "1". Sled agent comes online.
  • Sled 2 starts RSS, is configured to boot with a rack secret threshold of "1".
  • Sled 2 sees sled 1, advertised by ddmd. This RSS creates a plan that includes both sled 1 and 2 in the view of the "rack".
  • Sled 2's RSS sends a request to "start sled agent" to Sled 1. This request fails, because sled 1 already booted the sled agent by itself. (this was a very real error, seen here - thank you @iliana for noticing!).

Some ideas for mitigating:

  • As part of the bootstrap protocol, advertise whether or not the sled agent is already running. Ignore these already running sleds when configuring RSS. There's the possibility of raciness here (which machine gets picked up by which RSS?) but it makes this case more explicitly handled.
  • Make RSS generate a "rack UUID" - which we're doing for Nexus, anyway - that gets transmitted during the bootstrapping phases. This way, sleds can explicitly flag the error as "I'm already part of rack foo, but you're asking me to join rack bar.
  • Help a human make the right call - in a non-automated, production environment, we'll be presenting a list of sleds to an operator for confirmation. Perhaps we could figure out a way to use location information when doing this presentation - like, "you have two sleds that claim to be in the physical slot 1, what do you want to do about that?"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the related issue 1639 and the RSS bootstrap protocol described here, including how sleds are discovered through ddmd and inducted into a rack view. Compare the three proposed mitigation directions and define the expected behavior when separate RSS deployments share a network. Done means one deployment cannot mistakenly induct or start an already-running sled from another deployment, with the conflict handled explicitly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.