Azure / Azure/AgentBaker

node-exporter Error log Spam

Closed
#9,020 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
159
Forks
277
Avg merge
1d 16h
Merged PRs (30d)
178

Description

**What happened**: The service `node-exporter` is continuously writing ~3 error logs per minute.

**What you expected to happen**: No error logs.

**How to reproduce it**: For us, just spawn a new k8 node. We can observe this on every node.

Side note:
_I was surprised to find two node-exporter processes running on the AKS nodes. One is the built-in version from AgentBaker, and the other is the one we deployed ourselves via the `kube-prometheus-stack` DaemonSet.
What's your recommendation for scraping host metrics? Should we keep using our own node-exporter, or is it better to rely on the built-in one from AKS?_

**Environment**:

- AgentBaker version: `AKSUbuntu-2204gen2containerd-202605.14.0`
- Kubernetes version (use `kubectl version`): `Server Version: v1.34.2`
- OS (e.g. from /etc/os-release):
```
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```
- Kernel (e.g. `uname -a`): `Linux aks-xxx-yyyy-vmss000000 5.15.0-1111-azure #120-Ubuntu SMP Wed Apr 15 15:09:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux`

```
● node-exporter.service - Prometheus Node Exporter
Loaded: loaded (/etc/systemd/system/node-exporter.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2026-05-26 07:29:13 UTC; 1 month 27 days ago
Docs: https://github.com/prometheus/node_exporter
Main PID: 7782 (node-exporter)
Tasks: 6 (limit: 38497)
Memory: 26.8M
CPU: 42min 44.215s
CGroup: /system.slice/node-exporter.service
└─7782 /opt/bin/node-exporter --web.listen-address=10.240.11.6:19100 --no-collector.wifi --no-collector.hwmon --collector.cpu.info "--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run>

Jul 23 05:42:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:42:40.157Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
Jul 23 05:43:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:09.048Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:43:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:10.148Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:43:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:40.167Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:44:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:09.041Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
Jul 23 05:44:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:10.167Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:44:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:40.151Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:09.047Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:10.170Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:40.155Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
lines 1-21/21 (END)
```

```
Jul 23 05:57:10 aks-xxxx-yyyyy-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:57:10.165Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.6539e-05 err="error obtaining InfiniBand class info: read /sys/class/infiniband/mana_0/ports/1/rate: invalid argument"
```

Contributor guide

Open the contributing guide

Research direction

Start with the node-exporter.service status and repeated node-exporter-startup.sh logs, especially the collector.go:168 entries for the infiniband collector. Compare the built-in AgentBaker process with the kube-prometheus-stack DaemonSet and determine which exporter is intended to scrape host metrics. Done means the repeated error logs are addressed and the recommended exporter setup is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, linux, ubuntu
Domain
infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.