influxdata / influxdata/influxdb

deadman checks stopped being recognized by notification rule

Open
#23,118 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Recently about half our influxdb's stopped sending alerting for deadman checks. Based on the Check Status History the check is correctly marking the level as CRIT, however when checking the Noticiation Rules History, it is not reporting anything for the deadman check going to CRIT. All threshold alerts are working using the same Notification Rule which is set to alert for any check with a CRIT value

![image](https://user-images.githubusercontent.com/28870027/153301954-7c28c8e2-cf12-45de-9a30-a286a2888ff2.png)

zpool status is a threshold check which is alerting correctly
![image](https://user-images.githubusercontent.com/28870027/153302345-3441206a-642b-4ab3-a07e-4072f2d3cef8.png)

__Expected behavior:__
When state of Deadman check is CRIT, send an alert

__Actual behavior:__
Check state for the deadman goes to CRIT, however Noticiation Rule does not action the CRIT state of deadmans

__Environment info:__

```
influx version
Influx CLI 2.1.1 (git: none) build_date: 2022-02-09T22:40:04Z
influxd version
InfluxDB 2.1.1 (git: none) build_date: 2022-02-09T22:40:08Z
```
Check info
```
apiVersion: influxdata.com/v2alpha1
kind: CheckDeadman
metadata:
name: friendly-darwin-817001
spec:
every: 1m0s
level: CRIT
name: uptime deadman
query: |-
from(bucket: "fulldata")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "uptime_seconds")
|> filter(fn: (r) => r["_field"] == "value")
|> filter(fn: (r) => r["type"] == "numeric")
staleTime: 720h0m0s
status: active
statusMessageTemplate: ${r.host} is unreachable [${ r._check_name } = ${ r._level
}] (${r.dc})
timeSince: 5m0s
```
Notification rule
```
apiVersion: influxdata.com/v2alpha1
kind: NotificationRule
metadata:
name: xenodochial-maxwell-c17005
spec:
endpointName: pedantic-johnson-c17003
every: 1m0s
messageTemplate: 'Notification Rule: ${ r._notification_rule_name } triggered
by check: ${ r._check_name }: ${ r._message }'
name: tac monitoring pagerduty CRIT
statusRules:
- currentLevel: CRIT
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CheckDeadman and NotificationRule configuration shown in the issue, then compare Check Status History with Notification Rules History when the check reaches CRIT. Done means a CRIT deadman check triggers the configured notification, while the existing threshold-check notifications continue to work.

Written by the indexing model from the issue text.

Assessment

Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.