cloudnative-pg / cloudnative-pg/charts

Dollar sign in PrometheusRule template prevent low disk space alerts from firing

Open Beginner friendly
#966 1 comment 0 reactions 0 assignees View on GitHub
bug chart( cluster )
Dominant language
Go Template
Stars
621
Forks
265
Avg merge
7d 12h
Merged PRs (30d)
7

Description

A WAL storage volume for one of my cluster replicas ran out of disk space, but no alert was fired. When troubleshooting I found the following bug. Consider the following line:

https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/templates/prometheus-rule.yaml#L20

The dollar sign in this line is probably meant to match the end of a string. This works for most alerts, e.g.:

https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/prometheus_rules/cluster-high_connection-critical.yaml#L11

But for the low disk space alerts, the matching fails in in places where another string is being appended inside the label matcher (e.g. `-tbs` and `-wal`):

https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/prometheus_rules/cluster-low_disk_space-critical.yaml#L11-L12

After Helm inflation, the PromQL expression will then contain the label matchers (assuming the `cluster.fullname` is `cnpg-cluster`):

```promql
persistentvolumeclaim=~"cnpg-cluster-([1-9][0-9]*)$-tbs.*"
persistentvolumeclaim=~"cnpg-cluster-([1-9][0-9]*)$-wal"
```

The dollar sign causes the label not to match which, in turn, causes the alert rule not to fire.

Contributor guide

Open the contributing guide

Research direction

Start with charts/cluster/templates/prometheus-rule.yaml and compare its matcher pattern with charts/cluster/prometheus_rules/cluster-low_disk_space-critical.yaml. Render the Helm chart with cluster.fullname set to cnpg-cluster, then inspect the generated PromQL. Done means the -tbs and -wal PVC matchers match their intended volumes and the low disk space alerts can fire.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, prometheus
Domain
observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.