prometheus / prometheus/node_exporter

infiniband collector fails on MANA RDMA devices before device exclusion is applied

Open
#3,810 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system: output of uname -a

Observed on an Azure Ubuntu 22.04 VM using a Microsoft Azure Network Adapter (MANA). The public MANA documentation notes that Linux kernel 6.2 adds InfiniBand/RDMA support:

https://learn.microsoft.com/azure/virtual-network/accelerated-networking-mana-linux

The upstream driver is also under drivers/infiniband/hw/mana:

https://github.com/torvalds/linux/tree/master/drivers/infiniband/hw/mana

node_exporter version: output of node_exporter --version
node_exporter, version v1.12.1

This is the Microsoft Dalec-built node-exporter-kubernetes package from the upstream v1.12.1 source.

node_exporter command line flags

The collector is enabled by default. We also tried the new v1.12 device exclusion:

--collector.infiniband.device-exclude=^mana_.*
node_exporter log output
level=ERROR source=collector.go:168 msg="collector failed" name=infiniband err="error obtaining InfiniBand class info: read /sys/class/infiniband/mana_0/ports/1/rate: invalid argument"
Are you running node_exporter in Docker?

No. It runs directly as a systemd service on the VM.

What did you do that produced an error?

Scraped /metrics on a MANA-enabled Azure VM. MANA registers mana_0 under /sys/class/infiniband, but reading /sys/class/infiniband/mana_0/ports/1/rate returns EINVAL.

Adding --collector.infiniband.device-exclude=^mana_.* does not avoid the error. In v1.12.1, infinibandCollector.Update first calls c.fs.InfiniBandClass(), which fully parses every device, and applies c.deviceFilter.ignored(device.Name) only afterward. The failure therefore happens before the exclusion is evaluated.

Relevant ordering:

https://github.com/prometheus/node_exporter/blob/v1.12.1/collector/infiniband_linux.go

The underlying procfs parser treats rate as required and propagates its read error:

https://github.com/prometheus/procfs/blob/v0.21.1/sysfs/class_infiniband.go

What did you expect to see?

At minimum, the device exclusion should prevent node_exporter from parsing mana_*, allowing operators to avoid unsupported device attributes while retaining metrics for other HCAs on mixed-device nodes.

Ideally, procfs/node_exporter could also support MANA directly or tolerate an unavailable rate attribute if the remaining MANA RDMA data is useful.

What did you see instead?

The whole InfiniBand collector fails and reports:

node_scrape_collector_success{collector="infiniband"} 0

On mixed MANA and Mellanox nodes, the late filter means there is currently no configuration-only way to ignore MANA and retain collection from the other HCAs. The only working operator mitigation is --no-collector.infiniband, which disables all InfiniBand/RDMA metrics.

This resembles #3265 / prometheus/procfs#704, but that fix skips EINVAL for individual counter files. This failure is on the required port rate file and occurs before the v1.12 device filter is applied.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with collector/infiniband_linux.go and the InfiniBandClass entry point, then read procfs/sysfs/class_infiniband.go to understand where the required rate read fails. Verify the device-exclude path using the reported mana_0 sysfs layout; done means MANA can be excluded without failing collection for other HCAs on mixed-device nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability-sre, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.