kubernetes / kubernetes/node-problem-detector

scaling down will send unnecessary alerts

Open
#1,176 7 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
3.5k
Forks
702
Avg merge
1d 6h
Merged PRs (30d)
11

Description

The npd component checks the health status of kube-proxy via the /healthz interface
https://github.com/kubernetes/node-problem-detector/blob/2baac6c06751cc6dccd8b0ad9b6d08b6780f1a13/pkg/healthchecker/types/types.go#L83

When cluster-autoscaler performs elastic scaling down, it will add the ToBeDeletedByClusterAutoscaler taint to nodes. When a node has the ToBeDeletedByClusterAutoscaler taint, the /healthz interface of kube-proxy will consider itself to be in an unhealthy state.
https://github.com/kubernetes/kubernetes/blob/d4672d9a9db5687c4561048c4840ef1f8ccb3c75/pkg/proxy/healthcheck/proxy_health.go#L188

The elastic scaling down of cluster-autoscaler is standard behavior and should not trigger any alarms, as it can cause confusion.

Contributor guide

Open the contributing guide

Research direction

Read pkg/healthchecker/types/types.go at the kube-proxy /healthz check, then compare it with the linked Kubernetes proxy_health.go behavior for ToBeDeletedByClusterAutoscaler. Trace how npd turns that health result into an alert and verify that scaling down a tainted node no longer produces an unnecessary alarm.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.