canonical / canonical/cos-proxy-operator

Missing `juju_model` in alert_expr causes confusing alerts

Open
#182 0 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

Example nrpe alert rule:
```
name: CheckVaultHealthNrpeAlert
expr: avg_over_time(command_status{command="check_vault_health",juju_unit="vault/0"}[15m]) > 1 or (absent_over_time(command_status{command="check_vault_health",juju_unit="vault/0"}[10m]) == 1) or (absent_over_time(up{juju_unit="vault/0"}[10m]) == 1)
labels:
juju_application: vault
juju_model: kubernetes-dev
juju_unit: vault/0
nrpe_application: nrpe
nrpe_unit: nrpe/8
severity: {{ if eq $value 0.0 -}} info {{- else if eq $value 1.0 -}} warning {{- else if eq $value 2.0 -}} critical {{- else if eq $value 3.0 -}} error {{- end }}
annotations:
description: Check provided by nrpe_exporter in model {{ $labels.juju_model }} is failing.
Failing check = {{ $labels.command }}
Unit = {{ $labels.juju_unit }}
Value = {{ $value }}
Legend:
- StatusOK = 0
- StatusWarning = 1
- StatusCritical = 2
- StatusUnknown = 3
summary: Unit {{ $labels.juju_unit }}: {{ $labels.command }} {{ $labels.severity }}.
```

This alert rule will trigger an alert for vault/0 regardless of in what model it is failing.

Now, we have multiple models, e.g. kubernetes-prod with vault/0 deployed there as well. We had a case of vault/0 failing in kubernetes-dev but since these generic alert rules were created both for -dev and -prod, both have fired resulting in:
- multiple alerts for one failed units
- each alert added it's own juju_label model but the label was not matching the problematic unit -- instead it was the label of where cos-proxy that created this alert rule is located

### To Reproduce

model A:
vault/0 <-> nrpe <-> cos-proxy <---crm relation---> single COS prometheus

model B:
vault/0 <-> nrpe <-> cos-proxy <---crm relation---> the same COS prometheus

Vault is just an example, all NRPE alerts are affected with this kind of topology.

You'll see two identical NRPE rules created for both models so even if only one unit in one model fails, alerts for both models will fire, one with a "sensible" juju_model label, and the other one with the label of the wrong model.

### Environment

cos-proxy latest/stable 92

### Relevant log output

```shell
n/a - it's a design issue
```

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how cos-proxy creates NRPE alert rules for the topology described, then compare the duplicate rules and their juju_model labels across models. The issue does not identify files, tests, or a decided design; the work is done when one failed unit produces the correct model-specific alert without duplicate or misleading alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus, python
Domain
observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.