canonical / canonical/cos-proxy-operator
nrpe checks fail after adding relabel_configs in prometheus
- Dominant language
- Python
- Stars
- 3
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
After adding relabel_configs in cos-prometheus through prometheus-scrape-config-k8s charm, the nrpe checks fail. When the relabel_config is removed, the nrpe checks and alerts will work as expected.
### To Reproduce
1. Deploy the standard cos
2. Deploy nrpe charm in a separate model with relation to cos-proxy forwarding to cos-prometheus
3. Add the relabel_config to the cos-prometheus using the prometheus-scrape-config-k8s-charm.
relabel_configs: |-
- target_label: 'alerting_system'
replacement: 'nfvi-cos'
- target_label: 'site_identifier'
replacement: 'site1'
4. Observe that after the prometheus has resstarted and settled, the nrpe checks are failing.
### Environment
juju 3.4/stable, MAAS 3.4/stable
alertmanager alertmanager-k8s latest/edge 128
catalogue catalogue-k8s latest/stable 43
grafana grafana-k8s latest/stable
loki loki-k8s latest/stable
prometheus prometheus-k8s latest/stable
scrape-interval-config-monitors prometheus-scrape-config-k8s latest/edge
traefik traefik-k8s latest/stable 194
cos-proxy-monitors cos-proxy latest/edge 101
nrpe nrpe latest/stable
### Relevant log output
```shell
See the images in 'To Reproduce'
```
### Additional context
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the standard COS deployment with nrpe, cos-proxy, and prometheus-scrape-config-k8s, then compare the behavior with and without the shown relabel_configs. Start by inspecting the Prometheus and NRPE-related logs around the restart; done means NRPE checks and alerts continue working after the relabel configuration is applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100