oxidecomputer / oxidecomputer/hubris

Sidecar fan HSCs should be reset on fault

Open
#2,476 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⚠️ ereport fault-management
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

It looks like the hotswap controllers that switch power to the fans on Sidecar (e.g. U10) have no provision for auto-reset.

  • We have left the FAULT pin disconnected, which disables the chip's auto-recovery-after-delay behavior.
  • I have thus far not been able to find code in the firmware that would detect a channel fault (e.g. overcurrent) and take recovery action.
  • The FPGA is wired to some of the relevant signals, so there's a possibility it's doing recovery internally -- I've asked @Aaron-Hartwig to test on his sidecar.

In the event of an overcurrent detection on the fan, the HSC will cut power. If I'm right here, the power will not come back on without a hard power cycle -- and not an Ignition power cycle, either, because the HSCs are directly on the V54_BP rail, which can only be interrupted by disconnecting the bus bar input.

The Sidecar SP probably needs to take action to recover the rail in this case.

While we're in there, we should also make sure the Sidecar SP is appropriately reading PMBus fault status registers describing what happened, and produce e.g. an ereport.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Sidecar SP firmware handling for the fan hotswap controllers and the FPGA-connected fault signals described in the issue. Check how PMBus fault status registers are read and how ereports are produced; done means confirmed fault recovery and reporting behavior for an overcurrent event.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.