node-exporter daemonset keeps on creating new pods

Host operating system: output of uname -a

• Running as a pod on k8s cluster with ami type AL2_x86_64 and AL2_x86_64_GPU

node_exporter version: output of node_exporter --version

• node-exporter container has the following image: quay.io/prometheus/node-exporter:v1.7.0

node_exporter command line flags

• ARGS from the container

Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --path.udev.data=/host/root/run/udev/data
      --web.listen-address=[$(HOST_IP)]:9100

node_exporter log output

Are you running node_exporter in Docker?

• No, its being run as a part of prometheus helm chart v25.8.0

What did you do that produced an error?

• I simply created the node-exporter with default configuration, only adding tolerations of all of my nodegroups.

What did you expect to see?

• I expected to see one node-exporter pod per node, and a pod only gets spun up in case a new node comes up.

What did you see instead?

• I see loads of node-exporter pods being created on and on
• daemonset output:

k get daemonset -n monitoring -o wide
NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE    CONTAINERS      IMAGES                                    SELECTOR
prometheus-prometheus-node-exporter   12        12        12      12           12          kubernetes.io/os=linux   138m   node-exporter   quay.io/prometheus/node-exporter:v1.7.0   app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter

k describe daemonset prometheus-prometheus-node-exporter -n monitoring
Name:           prometheus-prometheus-node-exporter
Selector:       app.kubernetes.io/instance=prometheus,app.kubernetes.io/name=prometheus-node-exporter
Node-Selector:  kubernetes.io/os=linux
Labels:         app.kubernetes.io/component=metrics
                app.kubernetes.io/instance=prometheus
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=prometheus-node-exporter
                app.kubernetes.io/part-of=prometheus-node-exporter
                app.kubernetes.io/version=1.7.0
                helm.sh/chart=prometheus-node-exporter-4.24.0
Annotations:    deprecated.daemonset.template.generation: 1
                meta.helm.sh/release-name: prometheus
                meta.helm.sh/release-namespace: monitoring
                policies.kyverno.io/last-applied-patches:
                  autogen-tolerations-node-selectors.monitoring-tolerations-node-selectors.kyverno.io: added
                    /spec/template/spec/tolerations/2
Desired Number of Nodes Scheduled: 12
Current Number of Nodes Scheduled: 12
Number of Nodes Scheduled with Up-to-date Pods: 12
Number of Nodes Scheduled with Available Pods: 12
Number of Nodes Misscheduled: 0
Pods Status:  12 Running / 63 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=metrics
                    app.kubernetes.io/instance=prometheus
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=prometheus-node-exporter
                    app.kubernetes.io/part-of=prometheus-node-exporter
                    app.kubernetes.io/version=1.7.0
                    helm.sh/chart=prometheus-node-exporter-4.24.0
  Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: true
  Service Account:  prometheus-prometheus-node-exporter
  Containers:
   node-exporter:
    Image:      quay.io/prometheus/node-exporter:v1.7.0
    Port:       9100/TCP
    Host Port:  0/TCP
    Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --path.udev.data=/host/root/run/udev/data
      --web.listen-address=[$(HOST_IP)]:9100
    Liveness:   http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:9100/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      HOST_IP:  0.0.0.0
    Mounts:
      /host/proc from proc (ro)
      /host/root from root (ro)
      /host/sys from sys (ro)
  Volumes:
   proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
   sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:
   root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
Events:
  Type    Reason            Age                  From                  Message
  ----    ------            ----                 ----                  -------
  Normal  SuccessfulCreate  56m (x13 over 139m)  daemonset-controller  (combined from similar events): Created pod: prometheus-prometheus-node-exporter-jjkrl
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-twrkg
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-m5cc5
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-6t5ld
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-sdcb9
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-gmhlw
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-96b9z
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-98qnm
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-dwlbh
  Normal  SuccessfulCreate  56m                  daemonset-controller  Created pod: prometheus-prometheus-node-exporter-5rjrt

• Events from pods in Pending state

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  57m (x2 over 57m)   default-scheduler  0/13 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 1 Preemption is not helpful for scheduling, 12 No preemption victims found for incoming pod..
  Warning  FailedScheduling  51m (x4 over 57m)   default-scheduler  0/13 nodes are available: 13 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 13 No preemption victims found for incoming pod..
  Warning  FailedScheduling  46m                 default-scheduler  0/13 nodes are available: 1 node(s) were unschedulable, 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/13 nodes are available: 1 Preemption is not helpful for scheduling, 12 No preemption victims found for incoming pod..
  Warning  FailedScheduling  86s (x13 over 45m)  default-scheduler  0/12 nodes are available: 12 node(s) didn't have free ports for the requested pod ports. preemption: 0/12 nodes are available: 12 No preemption victims found for incoming pod..

• All the node-exporter pods in the namespace

k get pods -n monitoring -o wide | grep prometheus-prometheus-node-exporter
prometheus-prometheus-node-exporter-2tkt8        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-4hs56        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-57vxz        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-5m67m        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-5rjrt        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-68mlr        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-6mxvb        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-6t5ld        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-7qmtt        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-7t6wg        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-8f5hs        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-92qbc        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-96b9z        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-98qnm        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-9bh4p        1/1     Running   0          137m   10.8.136.64    ip-10-8-136-64.eu-west-1.compute.internal    <none>           <none>
prometheus-prometheus-node-exporter-9jkmc        1/1     Running   0          137m   10.8.148.75    ip-10-8-148-75.eu-west-1.compute.internal    <none>           <none>
prometheus-prometheus-node-exporter-9rw9s        1/1     Running   0          137m   10.8.161.180   ip-10-8-161-180.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-9svpj        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-bp6p8        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-c6nmg        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-c6ww9        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-ccjp5        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-cfqfj        1/1     Running   0          137m   10.8.154.220   ip-10-8-154-220.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-ckf58        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-cq664        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-d8tvf        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-dt95r        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-dwlbh        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-fx6xv        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-ggvvn        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-ghvbv        1/1     Running   0          137m   10.8.143.142   ip-10-8-143-142.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-gjls2        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-gmhlw        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-gxdzr        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-hx6q6        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-jg8g2        1/1     Running   0          137m   10.8.130.59    ip-10-8-130-59.eu-west-1.compute.internal    <none>           <none>
prometheus-prometheus-node-exporter-jjkrl        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-jntnv        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-jr44s        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-jtvll        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-k2bc6        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-k58jl        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-kdd6d        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-kj9x2        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-llj5g        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-lzj22        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-m5cc5        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-m7rwp        1/1     Running   0          137m   10.8.139.85    ip-10-8-139-85.eu-west-1.compute.internal    <none>           <none>
prometheus-prometheus-node-exporter-ms455        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-mxd5g        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-nwqrj        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-pfxf8        1/1     Running   0          137m   10.8.174.223   ip-10-8-174-223.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-qmmh6        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-rbh5r        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-rhf7v        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-rjq57        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-rmnth        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-rx6rz        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-s97fj        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-scgz9        1/1     Running   0          137m   10.8.153.113   ip-10-8-153-113.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-sdcb9        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-t4qnh        1/1     Running   0          137m   10.8.167.253   ip-10-8-167-253.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-t75lp        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-tdzjz        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-tp94n        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-tvrkc        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-tvw8t        1/1     Running   0          137m   10.8.156.134   ip-10-8-156-134.eu-west-1.compute.internal   <none>           <none>
prometheus-prometheus-node-exporter-twrkg        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-tzhls        0/1     Pending   0          54m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-vgr8p        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-vkdjb        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-vmqwf        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-wnlql        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-xjkcl        0/1     Pending   0          74m    <none>         <none>                                       <none>           <none>
prometheus-prometheus-node-exporter-ztdxt        1/1     Running   0          137m   10.8.172.16    ip-10-8-172-16.eu-west-1.compute.internal    <none>           <none>

Review the prometheus Helm chart linked in the report and the generated node-exporter DaemonSet, starting with its pod port, scheduling events, and configured arguments. Reproduce the default configuration with the reported tolerations; done would mean one running node-exporter pod per eligible node without accumulating Pending pods.