canonical / canonical/hardware-observer-operator
Flapping Alerts
- Dominant language
- Python
- Stars
- 20
- Forks
- 18
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 3
Description
Some of the alerts from this charm tend to be flapping a lot. The CollectorFailed alert is prone this, because it's very dependent on the hardware of the BMC, which tend to be flaky. This results in really annoying, flapping alerts, if the BMC doesn't respond in time.
Can we improve the alertrule by adding the `keep_firing_for` option and have it always at least fire for an hour?
From the [docs](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#defining-alerting-rules):
> There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met. This can be used to prevent situations such as flapping alerts, false resolutions due to lack of data loss, etc. Alerting rules without the keep_firing_for clause will deactivate on the first evaluation where the condition is not met (assuming any optional for duration described above has been satisfied).
Contributor guide
Research direction
Locate the CollectorFailed alert rule in the charm and review the linked Prometheus alerting-rules documentation first. Update the rule so keep_firing_for is set to one hour, then verify that the alert remains firing for that duration after its condition stops being met.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100