prometheus / prometheus/node_exporter

The slabinfo collector produces duplicate metrics on a host with Mellanox cards

Open
#3,506 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

The slabinfo collector in node_exporter 1.10.2 produces constant errors about duplicate metric names and values on a Linux host with Mellanox ConnectX-6 Ethernet and Infiniband cards. There are indeed slabs with duplicate names in the /proc/slabinfo file.

The error:

time=2025-12-17T06:08:16.966Z level=ERROR source=http.go:175 msg="error gathering metrics: 50 error(s) occurred:\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:736}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:776}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:736}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:776}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:736}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:776}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:396}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:396}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:736}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:776}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:352}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:352}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:736}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:44}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_ftes\"}  gauge:{value:8}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_active_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:84}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:84}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_object_size_bytes\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:776}} was collected before with the same name and label values\n* [from Gatherer #2[] collected metric \"node_slabinfo_objects_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:42}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_slabinfo_pages_per_slab\" { label:{name:\"slab\"  value:\"mlx5_fs_fgs\"}  gauge:{value:8}} was collected before with the same name and label values"

Some details about the system in question:

# cat /etc/os-release 
NAME="AlmaLinux"
VERSION="9.4 (Seafoam Ocelot)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.4 (Seafoam Ocelot)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
SUPPORT_END=2032-06-01

# uname -r
5.14.0-427.42.1.el9_4.x86_64

# lspci | grep -i mellanox
0000:19:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
0000:19:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
0000:41:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
0000:41:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
0000:9b:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
0000:9b:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]

# cat /proc/slabinfo | grep mlx
mlx5_fs_ftes          44     44    736   44    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_fgs           42     42    776   42    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_ftes          44     44    736   44    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_fgs           42     42    776   42    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_ftes          44     44    736   44    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_fgs           42     42    776   42    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_ftes          44     44    736   44    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_fgs           42     42    776   42    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_ftes         396    396    736   44    8 : tunables    0    0    0 : slabdata      9      9      0
mlx5_fs_fgs           42     42    776   42    8 : tunables    0    0    0 : slabdata      1      1      0
mlx5_fs_ftes         352    352    736   44    8 : tunables    0    0    0 : slabdata      8      8      0
mlx5_fs_fgs           84     84    776   42    8 : tunables    0    0    0 : slabdata      2      2      0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the slabinfo collector in node_exporter and reproduce the issue using the repeated entries shown from /proc/slabinfo. Confirm how duplicate slab names are handled; done means the collector no longer emits duplicate metric names and values on hosts with repeated Mellanox slab entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability-sre, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.