oxidecomputer / oxidecomputer/omicron
automate or expose recovery of nonfunctional sleds
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
With two switches and multiple instances of nexus, it should be possible to distinguish a single (or small number of) nonfunctioning sled -- no connectivity with sled-agent -- from a network partition. While no partition detection algorithm is ever foolproof, we should expose to operators at least the option to automatically or manually recover a sled that is clearly not working. The most reliable means of automatic recovery with capture of debugging information, short of on-site manual intervention, is to induce an NMI via the SP, presumably by communication via MGS. While policy is a bit fraught here -- this is disastrous if fully automated and we incorrectly diagnose a network partition as multiple faulty sleds -- mechanism is not. There is already a mechanism for NMI generation from pilot to faux-mgs to hubris. A reasonable starting place would simply be an API call that is plumbed through, without ever being executed automatically.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the existing NMI-generation path from pilot through faux-mgs to hubris, including how communication via MGS is represented. Define the proposed API boundary and verify that it can be invoked manually without automatic execution; done means the call is plumbed through and covered by the relevant existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100