canonical / canonical/hardware-observer-operator

Ipmitool provides false positive alarms - hardware Observer doesn't allow to suppress False Positive Alarms

Open
#539 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
20
Forks
18
Avg merge
1d 5h
Merged PRs (30d)
3

Description

### Description

We notice alarms being reported in COS dashboards as below. We have 15 servers of the same model and all the servers are reporting alerts for `SAS Mod Presence` and `OCP Mod Presence`

Image

From the BMC web console sensor readings are OK as below.

Image

It looks like ipmi-sensors output is interpreting it `Device Removed/Device Absent` as `Critical` in this case.

```
ubuntu@server:~$ sudo ipmi-sensors --output-sensor-state | grep -i 'SAS'
63 | SAS Mod Presence | Module/Board | Critical | N/A | N/A | 'Device Removed/Device Absent'
ubuntu@server:~$ sudo ipmi-sensors --output-sensor-state | grep -i 'OCP'
62 | OCP Mod Presence | Module/Board | Critical | N/A | N/A | 'Device Removed/Device Absent'
```
Additional outputs:
```
$ sudo ipmitool sdr elist | grep -E "Mod Presence"
OCP Mod Presence | 0Eh | ok | 44.1 | Device Absent
SAS Mod Presence | 0Fh | ok | 44.1 | Device Absent

$ sudo ipmitool sensor list | grep -i "mod"
OCP Mod Presence | 0x0 | discrete | 0x0180| na | na | na | na | na | na
SAS Mod Presence | 0x0 | discrete | 0x0180| na | na | na | na | na | na

```

Given the context, it seems, the servers (Wortmann TERRA SERVER 7120 G3 (Intel Xeon Gold 6242) with Intel Integrated BMC) in this case, seems to have additional slots for SAS and OCP modules, which are not in use. However, baseboard controller is equipped with sensors to detect the device presence and ipmitool is interpreting it as critical not to have the modules triggering a false positive alert in the environment. BMC web console seems to interpret it properly though.

Would it be possible to allow config options in the charm to mitigate such issues? Ask here is to allow certain level of alarm profiling to suppress such alarms once confirmed as false positive from the hardware-exporter.

### Charm revision

latest/stable

### Juju version

3.6.25

### Cloud

OpenStack

### Expected behaviour

When the device is absence is the expected behavior, we should be able to customize the alarm profiles.

### Reproduce / Test

This is specific to the server model and the base board as provided above.

### Notes & References

Partially related to [Issue 449](https://github.com/canonical/hardware-observer-operator/issues/449)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the charm's configuration and the behavior described for ipmi-sensors and ipmitool output, then read the related Issue 449. Define how alarm profiles should represent expected device absence and validate the result against the SAS Mod Presence and OCP Mod Presence examples, ensuring confirmed false positives can be suppressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.