influxdata / influxdata/influxdb
deadman checks stopped being recognized by notification rule
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Recently about half our influxdb's stopped sending alerting for deadman checks. Based on the Check Status History the check is correctly marking the level as CRIT, however when checking the Noticiation Rules History, it is not reporting anything for the deadman check going to CRIT. All threshold alerts are working using the same Notification Rule which is set to alert for any check with a CRIT value

zpool status is a threshold check which is alerting correctly

__Expected behavior:__
When state of Deadman check is CRIT, send an alert
__Actual behavior:__
Check state for the deadman goes to CRIT, however Noticiation Rule does not action the CRIT state of deadmans
__Environment info:__
```
influx version
Influx CLI 2.1.1 (git: none) build_date: 2022-02-09T22:40:04Z
influxd version
InfluxDB 2.1.1 (git: none) build_date: 2022-02-09T22:40:08Z
```
Check info
```
apiVersion: influxdata.com/v2alpha1
kind: CheckDeadman
metadata:
name: friendly-darwin-817001
spec:
every: 1m0s
level: CRIT
name: uptime deadman
query: |-
from(bucket: "fulldata")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "uptime_seconds")
|> filter(fn: (r) => r["_field"] == "value")
|> filter(fn: (r) => r["type"] == "numeric")
staleTime: 720h0m0s
status: active
statusMessageTemplate: ${r.host} is unreachable [${ r._check_name } = ${ r._level
}] (${r.dc})
timeSince: 5m0s
```
Notification rule
```
apiVersion: influxdata.com/v2alpha1
kind: NotificationRule
metadata:
name: xenodochial-maxwell-c17005
spec:
endpointName: pedantic-johnson-c17003
every: 1m0s
messageTemplate: 'Notification Rule: ${ r._notification_rule_name } triggered
by check: ${ r._check_name }: ${ r._message }'
name: tac monitoring pagerduty CRIT
statusRules:
- currentLevel: CRIT
```
Contributor guide
Research direction
Start by reproducing the CheckDeadman and NotificationRule configuration shown in the issue, then compare Check Status History with Notification Rules History when the check reaches CRIT. Done means a CRIT deadman check triggers the configured notification, while the existing threshold-check notifications continue to work.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100