bloomberg / bloomberg/goldpinger

Goldpinger is too sensitive to autoscaler activity

Open
#87 5 comments 5 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.7k
Forks
187
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

The order of operations for removing a node in Kubernetes is:

1. Cluster autoscaler removes VM from the underlying cloud provider
1. Node object enters NotReady state since the Kubelet process stops reporting
1. Cloud Controller eventually notices that the VM is gone and removes the Node object

The time between 2 and 3 can be quite long (many minutes in some clouds). Goldpinger continuously tries to reach the node during this time causing spikes in Goldpinger metrics.

**To Reproduce**
Steps to reproduce the behavior:

1. Overscale a cluster in a cloud with long deletion times such as Azure
1. Allow cluster to scale down
1. Observe peer failures in Goldpinger metrics and logs

**Expected behavior**

Goldpinger should provide a mechanism to filter out NotReady nodes from metric queries to focus on Nodes which are expected to be functioning normally.

**Screenshots**

Here's an example showing Goldpinger error rates spike as a cluster scaled down over a period of hours.

![Screen Shot 2020-05-26 at 4 50 07 PM](https://user-images.githubusercontent.com/4450708/82957742-014a7880-9f71-11ea-845b-ec8d684acb2b.png)

**Environment (please complete the following information):**
- Operating System and Version: N/A
- Browser [e.g. Firefox, Safari] (if applicable): N/A

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing how Goldpinger discovers Kubernetes nodes and queries peer metrics, then determine where NotReady nodes can be filtered. Done means autoscaler-related NotReady nodes no longer create peer failures or metric spikes while functioning nodes remain covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kubernetes, prometheus
Domain
devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.