oxidecomputer / oxidecomputer/omicron

automate or expose recovery of nonfunctional sleds

Open
#4,287 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

With two switches and multiple instances of nexus, it should be possible to distinguish a single (or small number of) nonfunctioning sled -- no connectivity with sled-agent -- from a network partition. While no partition detection algorithm is ever foolproof, we should expose to operators at least the option to automatically or manually recover a sled that is clearly not working. The most reliable means of automatic recovery with capture of debugging information, short of on-site manual intervention, is to induce an NMI via the SP, presumably by communication via MGS. While policy is a bit fraught here -- this is disastrous if fully automated and we incorrectly diagnose a network partition as multiple faulty sleds -- mechanism is not. There is already a mechanism for NMI generation from pilot to faux-mgs to hubris. A reasonable starting place would simply be an API call that is plumbed through, without ever being executed automatically.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the existing NMI-generation path from pilot through faux-mgs to hubris, including how communication via MGS is represented. Define the proposed API boundary and verify that it can be invoked manually without automatic execution; done means the call is plumbed through and covered by the relevant existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.