deckhouse / deckhouse/deckhouse
[extended-monitoring] node_exporter doesn't honor kubelet eviction thresholds for nodefs and other system mountpoints in GKE
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 162
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 653
Description
Preflight Checklist
- I agree to follow the Code of Conduct that this project adheres to.
- I have searched the issue tracker for an issue that matches the one I want to file, without success.
Version
v1.44.4
Expected Behavior
node_exporter should skip nodefs and other kubelet-related filesystems in accordance to Caution clause in Dechouse documentation from being exported.
Actual Behavior
node_exporter in GKE exports corresponding metrics causing Prometheus to alert about space usage in nodefs and other system mountpoints:
Node disk "" on mountpoint "/home/kubernetes/bin" is using more than 70% of storage capacity.
Node disk "" on mountpoint "/home/kubernetes/containerized_mounter" is using more than 70% of storage capacity.
Node disk "" on mountpoint "/home/containerd" is using more than 70% of storage capacity.
Node disk "" on mountpoint "/var/lib/kubelet" is using more than 70% of storage capacity.
Node disk "" on mountpoint "/home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet" is using more than 70% of storage capacity.
Steps To Reproduce
- Setup Deckhouse in GKE
- Fill
nodefsabove 70% - Look for alerts in Prometheus
Additional Information
This issue should possibly be fixed in node-exporter's DaemonSet by passing correct mountpoint for GKE.
Logs
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with modules/340-monitoring-kubernetes/templates/node-exporter/daemonset.yaml around line 81 and the linked extended-monitoring documentation. Reproduce the filesystem alerts in GKE and determine which kubelet-related mountpoints should be excluded. Done means node_exporter no longer exports those system mountpoints for alerts while retaining relevant node filesystem metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, kubernetes, prometheus
- Domain
- cloud, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100