[Bug]: 1.17 InfiniBandCharDeviceCheck marks nodes unhealthy due to missing /dev/infiniband/iss

Prerequisites

• I searched existing issues
• I can reproduce this issue

Code of Conduct

• I agree to follow NVSentinel's Code of Conduct

Bug Description

We are running on GB300 nodes which do not have /dev/infiniband/issm device nodes.

As far as I understand our provider decided to not create those devices. SM_DISABLED is set on ports. Since #1546 nvsentinel marks all of our GPU nodes as affected by this new InfiniBandCharDeviceCheck condition (cc @KaivalyaMDabhadkar @shaq918 ).

Error we see is Device mlx5_0 port 1: issm character device missing from /sys/class/infiniband_mad (expected for an InfiniBand-mode port); pods cannot open /dev/infiniband/issm*

Could you clarify why we decided to mark the lack of issm / Subnet Manager a fatal failure?

From our vantage point we could see that missing umad or uverbs is an obvious health concern, but we're not sure that issm per se is a problem: we certainly seem to have healthy deployments without issm.

Concrete question / suggestion: should the issm check be separate / opt-in, and perhaps not enabled by default? Checking for umad/uverbs sounds fine but issm -- a little ambiguous.

Component

Health Monitor

Steps to Reproduce

NVSentinel on nodes without issm devices

Environment

• NVSentinel version: 1.20.0
• Kubernetes version: 1.35.3
• Deployment method: Helm chart

Logs/Output

No response

The issue names InfiniBandCharDeviceCheck but no file or test path. Start by locating that check and reproducing it on nodes without /dev/infiniband/issm; compare its behavior with the umad/uverbs checks and SM_DISABLED. Done means the health result follows the project's intended policy for missing issm devices and the behavior is covered by an appropriate test.