GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine

config-reloader Restarts Multiple Times on Boot

Open
#472 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
232
Forks
109
Avg merge
1d 3h
Merged PRs (30d)
27

Description

I've been investigating why the `config-reloader` pod occasionally restarts multiple times while booting, and I've narrowed it down to [this line](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/3d1c0fbf4211bd34b955079297bece2e738da3c0/cmd/config-reloader/main.go#L91). I'm seeing the following in the logs:

```
Get \"http://localhost:19090/-/ready\": dial tcp 127.0.0.1:19090: connect: connection refused
```

Per the comment on line 73, it seems like the intent would be to continue polling in this situation, but instead the process exits and has to be restarted.

Contributor guide

Open the contributing guide

Research direction

Start in cmd/config-reloader/main.go, especially the polling comment around line 73 and the behavior at line 91. Reproduce the startup case with the readiness endpoint refusing connections, then run the relevant config-reloader tests or add coverage for that condition. Done means the process continues polling instead of exiting when localhost:19090 is not ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.