prometheus / prometheus/procfs

Export InfiniBandDevice parsing or add device filtering to InfiniBandClass

Open
#823 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
883
Forks
388
Avg merge
13h 28m
Merged PRs (30d)
3

Description

Problem

On NVIDIA B100+ systems, some ConnectX-7 NICs are NVSwitch/Fabric Manager-managed Limited PFs (VPD marked SMDL=SW_MNG). These NICs' hardware registers are owned by Fabric Manager, not the host. Reading any file under /sys/class/infiniband/<dev>/ports/*/counters/ triggers kernel ACCESS_REG(0x805) firmware errors:

mlx5_core 0000:xx:00.0: mlx5_cmd_out_err:834:(pid 8567): ACCESS_REG(0x805) op_mod(0x1) failed,status bad operation(0x2), syndrome (0x9a6171), err(-22)

9 lines per device per scrape, flooding dmesg. See node_exporter#3434 (https://github.com/prometheus/node_exporter/issues/3434) for full context.

Why InfiniBandClass() can't be used

InfiniBandClass() eagerly parses all devices — parseInfiniBandDevice → parseInfiniBandPort → parseInfiniBandCounters — in a single call. The caller has no opportunity to filter devices before counter files are read. By the time the caller iterates over the returned devices, the firmware errors have already been triggered.

The caller can identify FM-managed devices beforehand (by reading /sys/class/infiniband/<dev>/device/subsystem_device, which is world-readable), but cannot skip them because parseInfiniBandDevice is unexported.

Request

Either:

Option A: Export parseInfiniBandDevice (rename to InfiniBandDevice):

func (fs FS) InfiniBandDevice(name string) (*InfiniBandDevice, error)

This lets callers list the directory, filter, and parse per-device.

Option B: Add filtering to InfiniBandClass:

func (fs FS) InfiniBandClass(excludePattern ...string) (InfiniBandClass, error)

Option A is preferable — It is purely additive — InfiniBandClass() is unchanged, so all existing callers are unaffected. New users gain per-device parsing without any migration cost.

Workaround in node_exporter

We are currently using go:linkname to call the unexported parseInfiniBandDevice as a temporary bridge, which is fragile and not sustainable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing parseInfiniBandDevice, parseInfiniBandPort, parseInfiniBandCounters, and InfiniBandClass, then inspect how callers enumerate /sys/class/infiniband devices. Done means callers can filter devices before counters are read, while existing InfiniBandClass behavior remains unchanged; verify the API with the relevant InfiniBand parsing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, operating-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.