canonical / canonical/cos-proxy-operator

nrpe checks fail after adding relabel_configs in prometheus

Open
#157 2 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

After adding relabel_configs in cos-prometheus through prometheus-scrape-config-k8s charm, the nrpe checks fail. When the relabel_config is removed, the nrpe checks and alerts will work as expected.

### To Reproduce

1. Deploy the standard cos
2. Deploy nrpe charm in a separate model with relation to cos-proxy forwarding to cos-prometheus
3. Add the relabel_config to the cos-prometheus using the prometheus-scrape-config-k8s-charm.
relabel_configs: |-
- target_label: 'alerting_system'
replacement: 'nfvi-cos'
- target_label: 'site_identifier'
replacement: 'site1'
4. Observe that after the prometheus has resstarted and settled, the nrpe checks are failing.

### Environment

juju 3.4/stable, MAAS 3.4/stable

alertmanager alertmanager-k8s latest/edge 128
catalogue catalogue-k8s latest/stable 43
grafana grafana-k8s latest/stable
loki loki-k8s latest/stable
prometheus prometheus-k8s latest/stable
scrape-interval-config-monitors prometheus-scrape-config-k8s latest/edge
traefik traefik-k8s latest/stable 194

cos-proxy-monitors cos-proxy latest/edge 101
nrpe nrpe latest/stable

### Relevant log output

```shell
See the images in 'To Reproduce'
```

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the standard COS deployment with nrpe, cos-proxy, and prometheus-scrape-config-k8s, then compare the behavior with and without the shown relabel_configs. Start by inspecting the Prometheus and NRPE-related logs around the restart; done means NRPE checks and alerts continue working after the relabel configuration is applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.