bloomberg / bloomberg/goldpinger
Goldpinger is too sensitive to autoscaler activity
- Dominant language
- JavaScript
- Stars
- 2.7k
- Forks
- 187
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
The order of operations for removing a node in Kubernetes is:
1. Cluster autoscaler removes VM from the underlying cloud provider
1. Node object enters NotReady state since the Kubelet process stops reporting
1. Cloud Controller eventually notices that the VM is gone and removes the Node object
The time between 2 and 3 can be quite long (many minutes in some clouds). Goldpinger continuously tries to reach the node during this time causing spikes in Goldpinger metrics.
**To Reproduce**
Steps to reproduce the behavior:
1. Overscale a cluster in a cloud with long deletion times such as Azure
1. Allow cluster to scale down
1. Observe peer failures in Goldpinger metrics and logs
**Expected behavior**
Goldpinger should provide a mechanism to filter out NotReady nodes from metric queries to focus on Nodes which are expected to be functioning normally.
**Screenshots**
Here's an example showing Goldpinger error rates spike as a cluster scaled down over a period of hours.

**Environment (please complete the following information):**
- Operating System and Version: N/A
- Browser [e.g. Firefox, Safari] (if applicable): N/A
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by tracing how Goldpinger discovers Kubernetes nodes and queries peer metrics, then determine where NotReady nodes can be filtered. Done means autoscaler-related NotReady nodes no longer create peer failures or metric spikes while functioning nodes remain covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, kubernetes, prometheus
- Domain
- devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100