prometheus / prometheus/node_exporter
node_exporter not reporting filesystem metrics for additional volume mounts in Docker
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.8k
- Forks
- 2.7k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 8
Description
Host operating system: output of uname -a
Linux calixtoubuntuserver 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
build user: root@03d440803209
build date: 20240714-11:53:45
go version: go1.22.5
platform: linux/amd64
tags: unknown
node_exporter command line flags
--collector.filesystem
--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|run/.+|/snap/.+)($|/)
node_exporter log output
ts=2024-11-30T11:24:24.674Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)"
ts=2024-11-30T11:24:24.674Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.5, platform=linux/amd64, user=root@03d440803209, date=20240714-11:53:45, tags=unknown)"
ts=2024-11-30T11:24:24.677Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|run/.+|/snap/.+)($|/)
ts=2024-11-30T11:24:24.677Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-11-30T11:24:24.677Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-11-30T11:24:24.678Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-11-30T11:24:24.678Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=os
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=time
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-11-30T11:24:24.679Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-11-30T11:24:24.680Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-11-30T11:24:24.680Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100
Are you running node_exporter in Docker?
Yes
What did you do that produced an error?
- Mounted a volume /home/calixtoubuntu/Media:/media:ro in the node_exporter container
- The mount is visible inside the container in two places? When running
docker exec node_exporter df -h | grep media -A 1:
292.8G 144.0G 135.8G 51% /rootfs/media
/dev/mapper/ubuntu--vg-ubuntu--lv
--
292.8G 144.0G 135.8G 51% /media
/dev/mapper/ubuntu--vg-ubuntu--lv
- However, when querying Prometheus for filesystem metrics, the mount is not reported
What did you expect to see?
The filesystem metrics for the mounted volume at /media showing up in Prometheus queries like:
node_filesystem_avail_bytes{device="/dev/mapper/ubuntu--vg-ubuntu--lv", mountpoint="/media"}
What did you see instead?
Only the root filesystem metrics are reported:
node_filesystem_avail_bytes{device="/dev/mapper/ubuntu--vg-ubuntu--lv", fstype="ext4", instance="host.docker.internal:9100", job="node_exporter", mountpoint="/"} [value]
No metrics are reported for the /media mountpoint despite it being correctly mounted in the container.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the report with the shown Docker volume mount, filesystem collector flags, and df -h output, then compare the container mounts with the Prometheus filesystem query for /media. The work is done when the mounted /media filesystem is reported without incorrectly duplicating or omitting the root filesystem metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100