canonical / canonical/hardware-observer-operator
Ipmitool provides false positive alarms - hardware Observer doesn't allow to suppress False Positive Alarms
- Dominant language
- Python
- Stars
- 20
- Forks
- 18
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 3
Description
### Description
We notice alarms being reported in COS dashboards as below. We have 15 servers of the same model and all the servers are reporting alerts for `SAS Mod Presence` and `OCP Mod Presence`
From the BMC web console sensor readings are OK as below.
It looks like ipmi-sensors output is interpreting it `Device Removed/Device Absent` as `Critical` in this case.
```
ubuntu@server:~$ sudo ipmi-sensors --output-sensor-state | grep -i 'SAS'
63 | SAS Mod Presence | Module/Board | Critical | N/A | N/A | 'Device Removed/Device Absent'
ubuntu@server:~$ sudo ipmi-sensors --output-sensor-state | grep -i 'OCP'
62 | OCP Mod Presence | Module/Board | Critical | N/A | N/A | 'Device Removed/Device Absent'
```
Additional outputs:
```
$ sudo ipmitool sdr elist | grep -E "Mod Presence"
OCP Mod Presence | 0Eh | ok | 44.1 | Device Absent
SAS Mod Presence | 0Fh | ok | 44.1 | Device Absent
$ sudo ipmitool sensor list | grep -i "mod"
OCP Mod Presence | 0x0 | discrete | 0x0180| na | na | na | na | na | na
SAS Mod Presence | 0x0 | discrete | 0x0180| na | na | na | na | na | na
```
Given the context, it seems, the servers (Wortmann TERRA SERVER 7120 G3 (Intel Xeon Gold 6242) with Intel Integrated BMC) in this case, seems to have additional slots for SAS and OCP modules, which are not in use. However, baseboard controller is equipped with sensors to detect the device presence and ipmitool is interpreting it as critical not to have the modules triggering a false positive alert in the environment. BMC web console seems to interpret it properly though.
Would it be possible to allow config options in the charm to mitigate such issues? Ask here is to allow certain level of alarm profiling to suppress such alarms once confirmed as false positive from the hardware-exporter.
### Charm revision
latest/stable
### Juju version
3.6.25
### Cloud
OpenStack
### Expected behaviour
When the device is absence is the expected behavior, we should be able to customize the alarm profiles.
### Reproduce / Test
This is specific to the server model and the base board as provided above.
### Notes & References
Partially related to [Issue 449](https://github.com/canonical/hardware-observer-operator/issues/449)
Contributor guide
Research direction
Start by reviewing the charm's configuration and the behavior described for ipmi-sensors and ipmitool output, then read the related Issue 449. Define how alarm profiles should represent expected device absence and validate the result against the SAS Mod Presence and OCP Mod Presence examples, ensuring confirmed false positives can be suppressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100