prometheus / prometheus/node_exporter
node_filesystem collector unable deduplicate data from multihomed mounts like nfs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.8k
- Forks
- 2.7k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 8
Description
Host operating system: output of uname -a
3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
node_exporter, version 1.4.0 (branch: HEAD, revision: 7da1321761b3b8dfc9e496e1a60e6a476fec6018)
node_exporter command line flags
/usr/sbin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
node_exporter log output
Oct 21 19:34:07 node_exporter: level=error ts=2022-10-21T23:34:07.079Z caller=stdlib.go:105 msg="error gathering metrics: 7 error(s) occurred:\n* [from Gatherer #2] collected metric "node_filesystem_device_error" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_size_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.36870912e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_free_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_avail_bytes" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:5.2514783232e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048576e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_files_free" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:1.048568733e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric "node_filesystem_readonly" { label:<name:"device" value:"storagesystem:/exports/home" > label:<name:"fstype" value:"nfs" > label:<name:"mountpoint" value:"/home" > gauge:<value:0 > } was collected before with the same name and label values"
Are you running node_exporter in Docker?
No
What did you do that produced an error?
SAN has multiple A records for load balancing.
/proc/self/mounts
storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.1,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.1 0 0
storagesystem:/exports/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.1.1.2,mountvers=3,mountport=300,mountproto=tcp,local_lock=all,addr=1.1.1.2 0 0
What did you expect to see?
Deduplicate data from /proc based on local mounts, and iterate correctly.
What did you see instead?
Log errors above.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the node_filesystem collector and compare its handling of the two /proc/self/mounts entries shown in the report. Reproduce the duplicate metric errors with multihomed NFS mounts, then verify that mount data is deduplicated and iterated correctly without repeated metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability, operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100