Node Exporter leaving OPENFILE marker on pvc host paths

Host operating system: output of uname -a

Linux scnomgmt08 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

Including commented out flags that I've tried that also had no success

- --log.level=debug
- --collector.netstat.fields=.*
- --collector.filesystem.mount-points-exclude=^(.*)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
#- --no-collector.filesystem
#- --no-collector.diskstats
#- --no-collector.filefd
#- --no-collector.mountstats
#- --no-collector.xfs
#- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|var/lib/containerd/.+|host/root/.+|host/.+)($|/)
# copied from Anthos
#- --collector.filesystem.mount-points-exclude=^/(host/root/|host/)?(dev|proc|sys|var/lib/docker/.+)($|/)
#- --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$

node_exporter log output

ts=2023-09-06T22:48:07.504Z caller=filesystem_linux.go:55 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/kubelet/pods/385f109c-132c-4d61-bd6f-bdcc0ba345cf/volumes/kubernetes.io~portworx-volume/pvc-ca28e648-3a66-4bc9-ba8c-3d5d9b313406

Are you running node_exporter in Docker?

Kubernetes with containerd

What did you do that produced an error?

Pod with PVC attached is deleted  and gets stuck in terminating.  It will hang here forever until force deleted, node exporter cycled, or manual filesystem unmount

What did you expect to see?

Node exporter ignores the PVC filepath on the host system, according to the regex given.  

What did you see instead?

It appears that node exporter does not ignore the filepaths correctly, because it maintains an OPENFILE on the pvc mount path. Output from a custom script written to determine this:

CMD: node_exporter PID: 3087632 TID: dev OPENFILE: /host/root/var/lib/kubelet/pods/385f109c-132c-4d61-bd6f-bdcc0ba345cf/volumes/kubernetes.io~portworx-volume/pvc-ca28e648-3a66-4bc9-ba8c-3d5d9b313406 START_TIME: Fri Sep  1 04:57:05 2023 RUN_TIME: 05:18 CMDLINE: /bin/node_exporter--path.procfs=/host/proc--path.sysfs=/host/sys--path.rootfs=/host/root--web.listen-address=[0.0.0.0]:8999--log.level=debug--collector.netstat.fields=.*--collector.filesystem.mount-points-exclude=^(.*)--collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

Start by reading the filesystem collector at collectors/filesystem/filesystem_linux.go:55 and reviewing the supplied mount-points-exclude flags and debug output. Reproduce the PVC mount scenario if possible, then verify that the host PVC path is ignored and no OPENFILE remains after the pod is deleted; the issue provides no dedicated test to run.