oxidecomputer / oxidecomputer/hubris

ereports: How do we identify devices we send ereports about?

Open
#2,673 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

🤔 design control-plane-agent fault-management
Dominant language
Rust
Stars
3.6k
Forks
239
Avg merge
1d 12h
Merged PRs (30d)
23

Description

Talking with @hawkw in the context of #2639, we ran into an interesting case study:

  1. When the thermal loop detects that a fan is spinning too fast or too slow, we want to send an ereport to the host about it
  2. Ideally, we'd like to include sufficient information both for the fault management system, as well as operators and maintenance personnel to be well informed about what to do about that to resolve the issue

All good so far. However, a fan is not a single entity with a single refdes and name!

  • We observe (and control) fan speeds using a MAX31790 IC over I2C. The MAX31790 has a refdes (ex: "U58"), and the individual channel that we observe the failure on has a name (ex: "Northwest"). This IC contains multiple speed sensors (one for each fan rotor).
  • Specifically, each channel exposes a "speed"-type sensor through inventory/the sensor API.
  • A single Sanyo Denki "fan assembly" consists of two counter-rotating "fan rotors". I am unsure if the "fan assembly" has a unique refdes (or name).
  • For a compute sled (like cosmo), three "fan assemblies" are one field replaceable unit (FRU), called the "Sled Fan Tray". I am unsure if this has a specific refdes (or name).
  • The "Sled Fan Tray" contains an EEPROM containing vital product data (VPD).
  • The Sled Fan Tray EEPROM has a refdes (ex: "J34/U1") and a name (ex: "fan_vpd").

When we send the ereport, we certainly want to give the fault management system the refdes and name of the specific speed sensor when we tell it that a fan is "underspeed", so it knows to poll/monitor that specific fan for continued data to observe if the state changes.

However, refdes: "U58", name: "Northwest, is not something immediately useful or actionable to tell the operator, and neither is refdes: "J34/U1", name: "fan_vpd". On the cosmo, there is only a single fan tray, but on sidecar there are 4 "fan assemblies" that I believe can be individually hot-swapped (and are also CUSTOMER replaceable units (CRU) instead of FRUs).

https://rfd.shared.oxide.computer/rfd/0360#sec-hc-fans talks a bit about how we think about this from an fmtopo perspective, on a gimlet (nb: I think the all fans being fan=0 is a copy/paste error):

chassis=0
   fan-tray=0,serial=...
      fan=0,serial=...
         rotor=0
            sensor=0,class=remote-threshold,name=RPM
         rotor=1
            sensor=0,class=remote-threshold,name=RPM
      fan=0,serial=...
         rotor=0
            sensor=0,class=remote-threshold,name=RPM
         rotor=1
            sensor=0,class=remote-threshold,name=RPM
      fan=0,serial=...
         rotor=0
            sensor=0,class=remote-threshold,name=RPM
         rotor=1
            sensor=0,class=remote-threshold,name=RPM

And on a sidecar it is said:

The primary difference between Gimlet and Sidecar is that the Sidecar chassis would enumerate four bays, we would not enumerate the fan-tray, and the fans would also have indicators on them.

This leads me to believe that we might want to send "fan tray", "fan", and "rotor" in our messages, with "fan tray" being an Option and not present on sidecar?

Fans may be the most complex instance of "assembly vs entity that detects a problem", but even simpler temperature sensors are likely similar: we detect overheat from the sensor, but that is indicative of an overheat of a component, not a problem with the sensor itself. So, it seems good to discuss "best practice" on what to include in ereports!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the RFD 0360 fan-topology section and the context in #2639, then inspect the existing ereport and sensor API entry points. Determine how sensor, rotor, fan, and fan-tray identity should be represented across Gimlet and Sidecar, including temperature sensors. Done means a documented, agreed best practice for the ereport fields and topology relationships.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
embedded-iot, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.