node_filesystem_{free,avail}_bytes reporting values larger than node_filesystem_size_bytes

Host operating system: output of uname -a

$ uname -r
3.10.0-957.41.1.el7.x86_64

node_exporter version: output of node_exporter --version

$ node_exporter --version
node_exporter, version 1.0.0-rc.0 (branch: master, revision: a57f2465794ec60c40674706acc6c2ace12c1358)
  build user:       tdockendorf@pitzer-rw02.ten.osc.edu
  build date:       20200327-18:45:58
  go version:       go1.13.8

node_exporter command line flags

This is NFS root which produces lots of bind mounts so that is why we have a lot of filesystem ignores.

ExecStart=/usr/bin/node_exporter \
--collector.filesystem.ignored-fs-types=^(gpfs|nfs|nfs4|rootfs|tmpfs|cvmfs2|iso9660|autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ \
--collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
--collector.systemd.unit-whitelist=.+\.service \
--collector.systemd.unit-blacklist=(mmsdrserv)\.service \
--collector.netclass.ignored-devices=^(eth1|eth2|eth3|ib0|lo)$ \
--collector.filesystem.ignored-mount-points=^/(var/spool|var/log|var/lib/oprofile|var/account|var/cache/opensm|var/cache/ibutils|var/mmfs|var/adm/ras|var/lib/fail2ban|opt/puppetlabs/puppet/cache/clientbucket|opt/puppetlabs/puppet/cache/state|opt/dell/srvadmin/var|var/lib/identityfinder|var/lib/.identityfinder|var/cache/man|var/gdm|var/lib/xkb|var/lib/dbus|var/lib/nfs|var/lib/postfix|var/lib/gssproxy|var/singularity|var/lib/pcp/tmp|etc/lvm/cache|etc/lvm/archive|etc/lvm/backup|var/cache/foomatic|var/cache/logwatch|var/cache/httpd/ssl|var/cache/httpd/proxy|var/cache/php-pear|var/cache/systemtap|var/db/nscd|var/lib/dav|var/lib/dhcpd|var/lib/dhclient|var/lib/php|var/lib/pulse|var/lib/rsyslog|var/lib/ups|var/tmp|var/db/sudo|var/spool/cron|etc/sysconfig/iptables.d|etc/puppetlabs/mcollective|var/lib/node_exporter/textfile_collector|etc/adjtime|var/lib/arpwatch|var/lib/NetworkManager|var/cache/alchemist|var/lib/gdm|var/lib/iscsi|var/lib/ntp|var/lib/xen|var/empty/sshd/etc/localtime|var/lib/random-seed|var/lib/samba|etc/ofed-mic.map|opt/ipcm|usr/bin/turbostat|var/lib/pcp/pmdas/perfevent|var/lib/pcp/pmdas/infiniband|etc/sysconfig/network-scripts|etc/fstab|etc/pam.d|etc/security/access|etc/security/limits.d|etc/X11/xorg.conf.d|var/lib/sss|var/lib/logrotate||dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|cvmfs/.+|run/.+)($|/) \
--collector.systemd \
--collector.cpu.info \
--collector.mountstats \
--collector.ntp \
--no-collector.hwmon \
--no-collector.mdadm \
--no-collector.nfsd \
--no-collector.softnet \
--no-collector.thermal_zone \
--no-collector.zfs

Are you running node_exporter in Docker?

Not via Docker.

What did you do that produced an error?

Look at a graph in Grafana that uses these metrics. The filesystem avail bytes is an extremely large number and much larger than size bytes.

What did you expect to see?

I would never expect avail or free bytes for a filesystem to exceed the size.

What did you see instead?

The orange line is the avail bytes and the green line that appears to be near 0 is size in bytes.

The size in bytes is 879510155264 which is accurate but the avail bytes is so much larger the scale makes size in bytes look near zero.

Start with node_exporter’s filesystem collector, using the reported node_filesystem_free_bytes, node_filesystem_avail_bytes, and node_filesystem_size_bytes metrics and the NFS-root configuration as context. Reproduce the discrepancy on the described host and trace how these values are gathered; done means the values are correctly bounded or the filesystem-specific behavior is clearly documented and covered by a test.