Icinga / Icinga/icinga2

Option to retain acknowledgement through recovery

Open
#9,529 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement TBD
Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Hello,

We are using Icinga for disk usage monitoring and sometimes see a disk fluctuate a little bit around the warning mark. We will acknowledge the alert, but if the service recovers and then goes back to a warning stat, it loses the acknowledgement.

We tried using the "Sticky acknowledgement" option but it seems this only works for services going between Warn<->Crit state.

Using the "Downtime" option is not really preferred because that also prevents an alert when a service goes critical, which we do want.

Ideally the flow would be:

  1. Service goes from OK -> WARN
  2. We acknowledge the alert
  3. If the service recovers and then goes back to WARN, acknowledgement stays, indefinitely or until expire time is reached.
  4. If the service goes from WARN -> Crit remove the acknowledgement and re-alert.

This is because we will usually receive the alert, ACK it and set an expire time for a few days so people can fix it. If meanwhile it goes critical we would still like to know, but changes state between OK/NOT-OK for a bit it doesn´t really matter since we already know about the issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing acknowledgement, recovery, sticky-acknowledgement, and state-transition handling in the monitoring core. Compare the current Warn↔Crit behavior with the requested OK→Warn→OK→Warn and Warn→Crit flows; done means acknowledgements survive recovery until expiry but are removed when the service becomes critical.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.