canonical / canonical/hardware-observer-operator

Flapping Alerts

Open
#476 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
20
Forks
18
Avg merge
1d 5h
Merged PRs (30d)
3

Description

Some of the alerts from this charm tend to be flapping a lot. The CollectorFailed alert is prone this, because it's very dependent on the hardware of the BMC, which tend to be flaky. This results in really annoying, flapping alerts, if the BMC doesn't respond in time.

Can we improve the alertrule by adding the `keep_firing_for` option and have it always at least fire for an hour?
From the [docs](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#defining-alerting-rules):
> There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met. This can be used to prevent situations such as flapping alerts, false resolutions due to lack of data loss, etc. Alerting rules without the keep_firing_for clause will deactivate on the first evaluation where the condition is not met (assuming any optional for duration described above has been satisfied).

Contributor guide

Open the contributing guide

Research direction

Locate the CollectorFailed alert rule in the charm and review the linked Prometheus alerting-rules documentation first. Update the rule so keep_firing_for is set to one hour, then verify that the alert remains firing for that duration after its condition stops being met.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.