prometheus / prometheus/node_exporter

node_filesystem collector unable deduplicate data from multihomed mounts like nfs

Open
#2,514 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted bug
Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system: output of uname -a

3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.4.0 (branch: HEAD, revision: 7da1321761b3b8dfc9e496e1a60e6a476fec6018)

node_exporter command line flags

/usr/sbin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

node_exporter log output

Oct 21 19:34:07 node_exporter: level=error ts=2022-10-21T23:34:07.079Z caller=stdlib.go:105 msg="error gathering metrics: 7 error(s) occurred:\n* [from Gatherer #2] collected metric "node_filesystem_device_error" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_size_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.36870912e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_free_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_avail_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048576e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files_free" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048568733e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_readonly" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

SAN has multiple A records for load balancing.
/proc/self/mounts
storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.1,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.1 0 0
storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.2,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.2 0 0

What did you expect to see?

Deduplicate data from /proc based on local mounts, and iterate correctly.

What did you see instead?

Log errors above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the node_filesystem collector and compare its handling of the two /proc/self/mounts entries shown in the report. Reproduce the duplicate metric errors with multihomed NFS mounts, then verify that mount data is deduplicated and iterated correctly without repeated metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.