randomparity / randomparity/kdive
Capture BMC hardware event signals (SEL, thermtrip, prochot) when a System fails unexplained
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
Follow-on from the x86 half of #1816, answering its "could feed a pre-allocation health gate or
explain a mid-Run hang" scope item.
## Finding: IPMI is a strict superset of Redfish for the signals that explain a dead machine
Measured on a live AMI MegaRAC / AST2600 BMC:
| Source | Named sensors |
|---|---|
| Redfish `Chassis/Self/Thermal` (12 temps + 3 fans) + `Power` (15 voltages + 2 PSU) | 33 |
| IPMI `sdr elist` | 45 |
The 15 present only in IPMI:
```
CPU_THERMTRIP, CPU_PROCHOT, WATCHDOG2, PowerUnit, ChassisIntr,
PSU1/PSU2 Status, PSU1/PSU2 AC lost, PSU1/PSU2 IOUT, PSU1/PSU2 PIN, PSU1/PSU2 POUT
```
`CPU_THERMTRIP`, `CPU_PROCHOT` and `WATCHDOG2` are exactly the "why did this machine die" signals,
and they are **invisible through Redfish on this BMC**.
The System Event Log is separately useful and populated: 632 entries with timestamps, sensor names,
thresholds and actual readings, e.g.
```
06/21/2026 12:19:12 | Temperature X570 Temp | Upper Critical going high | Asserted | Reading 95 > Threshold 95 degrees C
```
## Two constraints any design must handle
1. **The SEL is mostly noise.** Of 382 entries sampled, 297 were one recurring chipset thermal
excursion and 82 were `Timestamp Clock Sync` events. A raw dump attached to a failed Run is
unreadable. The signal is the **diff against a baseline captured at allocation time**.
2. **It is slow to read.** ~6 entries/sec over IPMI LAN; the full 632-entry log takes ~100 s. That is
too slow for a synchronous teardown path and needs a bounded/incremental read with a stated
deadline (AGENTS.md five-part limit contract).
## Proposed scope
One PR:
- At allocation, record the SEL high-water mark and a health snapshot.
- On unexplained System failure (boot timeout, unreachable after crash, teardown failure), read only
entries past the mark and attach them to the Run as a diagnostic artifact.
- Surface `CPU_THERMTRIP`/`CPU_PROCHOT` explicitly — a thermally-tripped host should cordon
(`Resource.cordoned`) rather than return to the pool.
## Explicitly not proposed
Continuous polling as telemetry. `TelemetryService` on this BMC is an empty shell (0 MetricReports,
0 MetricReportDefinitions, **0 Triggers**, 3 generic MetricDefinitions), so there is no BMC-side
threshold push to subscribe to, and polling 45 sensors per Run buys little over reading the SEL on
failure.
## Depends on
The IPMI driver seam (#1821), since the useful signals are IPMI-only.
Contributor guide
Research direction
Start with AGENTS.md and the IPMI driver seam in #1821, then trace allocation, unexplained System failure, Run artifacts, and Resource.cordoned handling. Define how the allocation high-water mark and health snapshot flow into bounded SEL reads, and verify that post-mark diagnostics are attached while thermally tripped hosts are cordoned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100