randomparity / randomparity/kdive

Capture BMC hardware event signals (SEL, thermtrip, prochot) when a System fails unexplained

Open
#1,845 0 comments 0 reactions 0 assignees View on GitHub
area:control-retrieve effort:M priority:P3 provider:baremetal-bmc risk:daytime-only status:blocked type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

Follow-on from the x86 half of #1816, answering its "could feed a pre-allocation health gate or
explain a mid-Run hang" scope item.

## Finding: IPMI is a strict superset of Redfish for the signals that explain a dead machine

Measured on a live AMI MegaRAC / AST2600 BMC:

| Source | Named sensors |
|---|---|
| Redfish `Chassis/Self/Thermal` (12 temps + 3 fans) + `Power` (15 voltages + 2 PSU) | 33 |
| IPMI `sdr elist` | 45 |

The 15 present only in IPMI:

```
CPU_THERMTRIP, CPU_PROCHOT, WATCHDOG2, PowerUnit, ChassisIntr,
PSU1/PSU2 Status, PSU1/PSU2 AC lost, PSU1/PSU2 IOUT, PSU1/PSU2 PIN, PSU1/PSU2 POUT
```

`CPU_THERMTRIP`, `CPU_PROCHOT` and `WATCHDOG2` are exactly the "why did this machine die" signals,
and they are **invisible through Redfish on this BMC**.

The System Event Log is separately useful and populated: 632 entries with timestamps, sensor names,
thresholds and actual readings, e.g.

```
06/21/2026 12:19:12 | Temperature X570 Temp | Upper Critical going high | Asserted | Reading 95 > Threshold 95 degrees C
```

## Two constraints any design must handle

1. **The SEL is mostly noise.** Of 382 entries sampled, 297 were one recurring chipset thermal
excursion and 82 were `Timestamp Clock Sync` events. A raw dump attached to a failed Run is
unreadable. The signal is the **diff against a baseline captured at allocation time**.
2. **It is slow to read.** ~6 entries/sec over IPMI LAN; the full 632-entry log takes ~100 s. That is
too slow for a synchronous teardown path and needs a bounded/incremental read with a stated
deadline (AGENTS.md five-part limit contract).

## Proposed scope

One PR:

- At allocation, record the SEL high-water mark and a health snapshot.
- On unexplained System failure (boot timeout, unreachable after crash, teardown failure), read only
entries past the mark and attach them to the Run as a diagnostic artifact.
- Surface `CPU_THERMTRIP`/`CPU_PROCHOT` explicitly — a thermally-tripped host should cordon
(`Resource.cordoned`) rather than return to the pool.

## Explicitly not proposed

Continuous polling as telemetry. `TelemetryService` on this BMC is an empty shell (0 MetricReports,
0 MetricReportDefinitions, **0 Triggers**, 3 generic MetricDefinitions), so there is no BMC-side
threshold push to subscribe to, and polling 45 sensors per Run buys little over reading the SEL on
failure.

## Depends on

The IPMI driver seam (#1821), since the useful signals are IPMI-only.

Contributor guide

Open the contributing guide

Research direction

Start with AGENTS.md and the IPMI driver seam in #1821, then trace allocation, unexplained System failure, Run artifacts, and Resource.cordoned handling. Define how the allocation high-water mark and health snapshot flow into bounded SEL reads, and verify that post-mark diagnostics are attached while thermally tripped hosts are cordoned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.