influxdata / influxdata/influxdb

[bug] Notification rule ANY to CRIT not working

Open
#21,854 0 comments 3 reactions 0 assignees View on GitHub
area/2.x monitoring-and-alerting
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
1. Add Check (for every minute) and Notification endpoint
Screenshot 2021-07-15 at 12 23 03

2. Add Notification rule `Critical` with condition: status changes from `ANY` to `CRIT`
Screenshot 2021-07-15 at 12 17 02

3. Add Notification rule `Alive` with condition: status changes from `ANY` to `OK`
Screenshot 2021-07-15 at 12 20 44

4. Stop `Telegraf` on monitored host
5. Wait for 2 minutes
6. Run `Telegraf` on monitored host
7. Wait for 2 minutes

__Expected behavior:__
Received 2 messages:
```
Host status is `crit`
Host status is `ok`
```

__Actual behavior:__
Received 1 message:
```
Host status is `ok`
```

__Environment info:__
* System info: `Linux 4.9.0-16-amd64 x86_64`
* InfluxDB version: `InfluxDB 2.0.3 (git: fe04d346df) build_date: 2020-12-15T01:00:16Z`

__Config:__
default

__Alert History:__
Checks:
Screenshot 2021-07-15 at 12 33 30

Notifications:
Screenshot 2021-07-15 at 12 33 59

__Conclusion:__
Something wrong happens with detecting `crit` -> `ok` transition

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the notification-rule scenario with a per-minute Check, a Notification endpoint, and Telegraf stopped and restarted as described. Trace how transitions from ANY to CRIT and from ANY to OK are evaluated, then use the Check and Notification alert histories to verify that both expected messages are delivered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.