bloomberg / bloomberg/goldpinger

When PING_NUMBER is nonzero, there are many nodes that are immediately marked as unhealthy

Open
#97 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.7k
Forks
187
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
If I start off with 50 healthy nodes and then set PING_NUMBER to 20, I notice that roughly 30 nodes get marked as unhealthy. It appears that 20 nodes ping 20 nodes instead of 50 (all) nodes pinging 20 nodes. When I unset PING_NUMBER, the nodes go back to being healthy.

I think the bug is [here](https://github.com/bloomberg/goldpinger/blob/ca676bcbc2da02fdd5b796689ea727896ad15c74/pkg/goldpinger/client.go#L54). It seems to me that `AllPods()` should be "checked" but only `SelectPods()` are actually being checked which results in `SelectPods()` pinging `SelectPods()`.

**Expected behavior**
All nodes ping PING_NUMBER nodes and no nodes are marked as unhealthy as a result of applying change to set PING_NUMBER.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/goldpinger/client.go at the referenced line and compare how AllPods() and SelectPods() are used when PING_NUMBER is 20. Reproduce with 50 healthy nodes, then verify that all nodes ping the selected nodes and remain healthy after the setting changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.