oxidecomputer / oxidecomputer/hubris
Consider sending an `ereport` when compute sled host CPUs appear to hang
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
We can currently observe a "sign of life" in gimlet and cosmo compute sleds, as the host software toggles a GPIO, and the FPGA both counts the number of toggles, as well as measures the time in some kind of unit since the last toggle was observed. This is exposed upstack in component details of the CPU.
In cases where the Host CPU does hang, this toggling stops, and the timer then exceeds a level that we would expect. Since our sequencer tasks already periodically monitor various information, it would be possible to also check the toggle timer value ourselves, and issue an ereport when the threshold exceeds some expected amount with tolerance, informing upstack fault management that it may be necessary to perform corrective action, such as triggering an NMI, or even power-cycling the sled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the sequencer tasks that periodically monitor sled information and the component-details path exposing the CPU toggle timer. Determine the expected threshold and tolerance, then trace how an ereport reaches upstack fault management. Done means a host-CPU hang condition emits the intended ereport without false positives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100