oxidecomputer / oxidecomputer/hubris
ereports: How do we identify devices we send ereports about?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
Talking with @hawkw in the context of #2639, we ran into an interesting case study:
- When the thermal loop detects that a fan is spinning too fast or too slow, we want to send an ereport to the host about it
- Ideally, we'd like to include sufficient information both for the fault management system, as well as operators and maintenance personnel to be well informed about what to do about that to resolve the issue
All good so far. However, a fan is not a single entity with a single refdes and name!
- We observe (and control) fan speeds using a MAX31790 IC over I2C. The MAX31790 has a refdes (ex: "U58"), and the individual channel that we observe the failure on has a name (ex: "Northwest"). This IC contains multiple speed sensors (one for each fan rotor).
- Specifically, each channel exposes a "speed"-type sensor through inventory/the sensor API.
- A single Sanyo Denki "fan assembly" consists of two counter-rotating "fan rotors". I am unsure if the "fan assembly" has a unique refdes (or name).
- For a compute sled (like cosmo), three "fan assemblies" are one field replaceable unit (FRU), called the "Sled Fan Tray". I am unsure if this has a specific refdes (or name).
- The "Sled Fan Tray" contains an EEPROM containing vital product data (VPD).
- The Sled Fan Tray EEPROM has a refdes (ex: "J34/U1") and a name (ex: "fan_vpd").
When we send the ereport, we certainly want to give the fault management system the refdes and name of the specific speed sensor when we tell it that a fan is "underspeed", so it knows to poll/monitor that specific fan for continued data to observe if the state changes.
However, refdes: "U58", name: "Northwest, is not something immediately useful or actionable to tell the operator, and neither is refdes: "J34/U1", name: "fan_vpd". On the cosmo, there is only a single fan tray, but on sidecar there are 4 "fan assemblies" that I believe can be individually hot-swapped (and are also CUSTOMER replaceable units (CRU) instead of FRUs).
https://rfd.shared.oxide.computer/rfd/0360#sec-hc-fans talks a bit about how we think about this from an fmtopo perspective, on a gimlet (nb: I think the all fans being fan=0 is a copy/paste error):
chassis=0
fan-tray=0,serial=...
fan=0,serial=...
rotor=0
sensor=0,class=remote-threshold,name=RPM
rotor=1
sensor=0,class=remote-threshold,name=RPM
fan=0,serial=...
rotor=0
sensor=0,class=remote-threshold,name=RPM
rotor=1
sensor=0,class=remote-threshold,name=RPM
fan=0,serial=...
rotor=0
sensor=0,class=remote-threshold,name=RPM
rotor=1
sensor=0,class=remote-threshold,name=RPM
And on a sidecar it is said:
The primary difference between Gimlet and Sidecar is that the Sidecar chassis would enumerate four bays, we would not enumerate the fan-tray, and the fans would also have indicators on them.
This leads me to believe that we might want to send "fan tray", "fan", and "rotor" in our messages, with "fan tray" being an Option and not present on sidecar?
Fans may be the most complex instance of "assembly vs entity that detects a problem", but even simpler temperature sensors are likely similar: we detect overheat from the sensor, but that is indicative of an overheat of a component, not a problem with the sensor itself. So, it seems good to discuss "best practice" on what to include in ereports!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the RFD 0360 fan-topology section and the context in #2639, then inspect the existing ereport and sensor API entry points. Determine how sensor, rotor, fan, and fan-tray identity should be represented across Gimlet and Sidecar, including temperature sensors. Done means a documented, agreed best practice for the ereport fields and topology relationships.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100