prometheus / prometheus/node_exporter

Limit which "pfs" when querying mellanox cards

Open
#3,434 5 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

https://docs.nvidia.com/dgx/dgx-os-7-user-guide/known_issues.html#id16

With this note:

Accessing PF0 and PF1 is restricted. Currently, there is no temporary solution available.

[11176.517416] mlx5_core 0000:05:00.0: mlx5_cmd_out_err:835:(pid 18360): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.534892] mlx5_core 0000:05:00.0: mlx5_cmd_out_err:835:(pid 18360): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.589308] mlx5_core 0000:05:00.1: mlx5_cmd_out_err:835:(pid 10354): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.607052] mlx5_core 0000:05:00.1: mlx5_cmd_out_err:835:(pid 10354): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)

I didn't find a github issue about this in here. Maybe there could be a way to limit which PFs node_exporter tries to get data about?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported ACCESS_REG errors on a system with Mellanox cards and inspect how node_exporter queries PFs. Determine where PF selection is handled and define the configuration or filtering behavior needed to skip restricted PF0 and PF1. Done means the exporter no longer triggers these errors while still collecting data from permitted PFs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability, operating-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.