oxidecomputer / oxidecomputer/hubris
Sidecar fan HSCs should be reset on fault
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
It looks like the hotswap controllers that switch power to the fans on Sidecar (e.g. U10) have no provision for auto-reset.
- We have left the FAULT pin disconnected, which disables the chip's auto-recovery-after-delay behavior.
- I have thus far not been able to find code in the firmware that would detect a channel fault (e.g. overcurrent) and take recovery action.
- The FPGA is wired to some of the relevant signals, so there's a possibility it's doing recovery internally -- I've asked @Aaron-Hartwig to test on his sidecar.
In the event of an overcurrent detection on the fan, the HSC will cut power. If I'm right here, the power will not come back on without a hard power cycle -- and not an Ignition power cycle, either, because the HSCs are directly on the V54_BP rail, which can only be interrupted by disconnecting the bus bar input.
The Sidecar SP probably needs to take action to recover the rail in this case.
While we're in there, we should also make sure the Sidecar SP is appropriately reading PMBus fault status registers describing what happened, and produce e.g. an ereport.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Sidecar SP firmware handling for the fan hotswap controllers and the FPGA-connected fault signals described in the issue. Check how PMBus fault status registers are read and how ereports are produced; done means confirmed fault recovery and reporting behavior for an overcurrent event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100