prometheus / prometheus/node_exporter

netclass collector fails while trying to get metrics from ignored devices

Open
#1,915 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted bug
Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system: output of uname -a

Linux hostname 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version
node_exporter, version 1.0.1 (branch: HEAD, revision: 3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)
  build user:       root@1f76dbbcfa55
  build date:       20200616-12:44:12
  go version:       go1.14.4
node_exporter command line flags
--collector.textfile.directory=/var/lib/node_exporter/textfile_collector
--collector.ntp
--collector.processes
--collector.netclass.ignored-devices='^veth.*'
--collector.netdev.device-blacklist='^veth.*'
Are you running node_exporter in Docker?

no

What did you do that produced an error?

Docker removed veth interface and that triggered netclass scrape collector failure

What did you expect to see?

I would expect ignored devices not to trigger netclass scrape collector failure (veth.* in this case).
In fact node_exporter shouldn't even attempt to get metrics for these devices as that is useless.

What did you see instead?
# cat /var/log/syslog | grep "veth67506fc"
Dec 17 05:15:57 hostname systemd-udevd[44530]: Could not generate persistent MAC address for veth67506fc: No such file or directory
Dec 17 05:15:58 hostname kernel: [9799969.809182] eth1: renamed from veth67506fc
Dec 17 05:15:58 hostname node_exporter[6459]: level=error ts=2020-12-17T05:15:58.486Z caller=collector.go:161 msg="collector failed" name=netclass duration_seconds=0.044157776 err="could not get net class info: error obtaining net class info: open /sys/class/net/veth67506fc: no such file or directory"
Dec 17 05:15:59 hostname kernel: [9799970.764950] veth67506fc: renamed from eth1

Metrics for these devices are not exposed by node_exporter, as expected:

$ ifconfig | grep veth
veth06454cb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth09549bc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth139b970: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth18767aa: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth1ce105e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth2a93f53: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth2f43cdf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth3ea67ac: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth52036bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth68de7c9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth78836e8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth791ccd1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
veth79cf6f7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
vethfb80213: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
$ curl -s localhost:9100/metrics | grep veth
$

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the netclass collector and the --collector.netclass.ignored-devices flag, then reproduce the scrape while an ignored veth device is removed. Done means ignored devices are not queried and their disappearance does not cause the netclass collector to fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.