Option to retain acknowledgement through recovery
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 616
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 10
Description
Hello,
We are using Icinga for disk usage monitoring and sometimes see a disk fluctuate a little bit around the warning mark. We will acknowledge the alert, but if the service recovers and then goes back to a warning stat, it loses the acknowledgement.
We tried using the "Sticky acknowledgement" option but it seems this only works for services going between Warn<->Crit state.
Using the "Downtime" option is not really preferred because that also prevents an alert when a service goes critical, which we do want.
Ideally the flow would be:
- Service goes from OK -> WARN
- We acknowledge the alert
- If the service recovers and then goes back to WARN, acknowledgement stays, indefinitely or until expire time is reached.
- If the service goes from WARN -> Crit remove the acknowledgement and re-alert.
This is because we will usually receive the alert, ACK it and set an expire time for a few days so people can fix it. If meanwhile it goes critical we would still like to know, but changes state between OK/NOT-OK for a bit it doesn´t really matter since we already know about the issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing acknowledgement, recovery, sticky-acknowledgement, and state-transition handling in the monitoring core. Compare the current Warn↔Crit behavior with the requested OK→Warn→OK→Warn and Warn→Crit flows; done means acknowledgements survive recovery until expiry but are removed when the service becomes critical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100