cloudnative-pg / cloudnative-pg/charts
Dollar sign in PrometheusRule template prevent low disk space alerts from firing
- Dominant language
- Go Template
- Stars
- 621
- Forks
- 265
- Avg merge
- 7d 12h
- Merged PRs (30d)
- 7
Description
A WAL storage volume for one of my cluster replicas ran out of disk space, but no alert was fired. When troubleshooting I found the following bug. Consider the following line:
https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/templates/prometheus-rule.yaml#L20
The dollar sign in this line is probably meant to match the end of a string. This works for most alerts, e.g.:
https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/prometheus_rules/cluster-high_connection-critical.yaml#L11
But for the low disk space alerts, the matching fails in in places where another string is being appended inside the label matcher (e.g. `-tbs` and `-wal`):
https://github.com/cloudnative-pg/charts/blob/b21ad2dac7992215b8572007adb7448e3c8cc5db/charts/cluster/prometheus_rules/cluster-low_disk_space-critical.yaml#L11-L12
After Helm inflation, the PromQL expression will then contain the label matchers (assuming the `cluster.fullname` is `cnpg-cluster`):
```promql
persistentvolumeclaim=~"cnpg-cluster-([1-9][0-9]*)$-tbs.*"
persistentvolumeclaim=~"cnpg-cluster-([1-9][0-9]*)$-wal"
```
The dollar sign causes the label not to match which, in turn, causes the alert rule not to fire.
Contributor guide
Research direction
Start with charts/cluster/templates/prometheus-rule.yaml and compare its matcher pattern with charts/cluster/prometheus_rules/cluster-low_disk_space-critical.yaml. Render the Helm chart with cluster.fullname set to cnpg-cluster, then inspect the generated PromQL. Done means the -tbs and -wal PVC matchers match their intended volumes and the low disk space alerts can fire.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100