prometheus / prometheus/node_exporter
Limit which "pfs" when querying mellanox cards
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.8k
- Forks
- 2.7k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 8
Description
https://docs.nvidia.com/dgx/dgx-os-7-user-guide/known_issues.html#id16
With this note:
Accessing PF0 and PF1 is restricted. Currently, there is no temporary solution available.
[11176.517416] mlx5_core 0000:05:00.0: mlx5_cmd_out_err:835:(pid 18360): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.534892] mlx5_core 0000:05:00.0: mlx5_cmd_out_err:835:(pid 18360): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.589308] mlx5_core 0000:05:00.1: mlx5_cmd_out_err:835:(pid 10354): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
[11176.607052] mlx5_core 0000:05:00.1: mlx5_cmd_out_err:835:(pid 10354): ACCESS_REG(0x805) op_mod(0x1) failed, status bad operation(0x2), syndrome (0x9a6171), err(-22)
I didn't find a github issue about this in here. Maybe there could be a way to limit which PFs node_exporter tries to get data about?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported ACCESS_REG errors on a system with Mellanox cards and inspect how node_exporter queries PFs. Determine where PF selection is handled and define the configuration or filtering behavior needed to skip restricted PF0 and PF1. Done means the exporter no longer triggers these errors while still collecting data from permitted PFs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- observability, operating-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100