oxidecomputer / oxidecomputer/hubris

Consider sending an `ereport` when compute sled host CPUs appear to hang

Open
#2,658 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⚠️ ereport fault-management
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

We can currently observe a "sign of life" in gimlet and cosmo compute sleds, as the host software toggles a GPIO, and the FPGA both counts the number of toggles, as well as measures the time in some kind of unit since the last toggle was observed. This is exposed upstack in component details of the CPU.

In cases where the Host CPU does hang, this toggling stops, and the timer then exceeds a level that we would expect. Since our sequencer tasks already periodically monitor various information, it would be possible to also check the toggle timer value ourselves, and issue an ereport when the threshold exceeds some expected amount with tolerance, informing upstack fault management that it may be necessary to perform corrective action, such as triggering an NMI, or even power-cycling the sled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the sequencer tasks that periodically monitor sled information and the component-details path exposing the CPU toggle timer. Determine the expected threshold and tolerance, then trace how an ereport reaches upstack fault management. Done means a host-CPU hang condition emits the intended ereport without false positives.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.