Notebooks(Alerts): Deadman definition sends notifications even when stream is healthy
- Dominant language
- TypeScript
- Stars
- 117
- Forks
- 51
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 4
Description
## About the bug
**Steps to reproduce:**
List the minimal actions needed to reproduce the behavior.
1. Setup an echo listener to receive HTTP messages
2. Go to Influxdbv2 UI (note that during this test the feature flag 'notebooksNewEndpoints' was set)
3. Add test data
4. Create an Alert Notebook
5. Using the test data define the query to retrieve a numeric stream
6. In the alert section define a trigger using the notification category "missing for longer than".
7. Set values for the time frame and cutoff that will fall within the test data (e.g. 5m timeframe and 15m cutoff).
8. Set the every value to match the time frame (e.g. 5m)
9. Select HTTP integration and use the URL for the listener in step 1
10. Export the notebook as an Alert Task.
11. Monitor messages sent to the listener
**Expected behavior:**
With a "deadman" alert, expect for notifications to arrive once the stream of test data comes to an end, and for only 1 notification to be sent per expiration during the 'every' period, given that in the time frame there are no points in the stream.
**Actual behavior:**
I'm seeing notifications with for example `_level: OK` coming from the created task. Also generating spam #3707.
I compared this with a classic confguration; deadman check, notification endpoint, notification rule, and this sent 1 notification when the data stream terminated and no notifications before then.
**Visual Proof:**
the definition

console messages from the listener

console messages from the listener with standard deadman+endpoint+rule

## About your environment
**Environment info:**
Testing in K8SIDPE remocal
latest commmit in project UI
```
commit 22850c6b4a45fa63c8e8cf267439f90cc8e58460 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Author: Jakub Bednář
Date: Mon Mar 7 08:19:59 2022 +0100
```
latest commit in project K8SIDPE
```
commit e0ea3f9c0f3f5e186077e7cbe3aa399ba8abe317 (HEAD -> master, origin/master, origin/HEAD)
Author: wiedld
Date: Mon Mar 7 17:03:34 2022 -0800
```
## BTW
I see that the flux script for the generated task is using `|> monitor["check"](data: check, messageFn: messageFn, crit: trigger)`.
However the monitor api has an explicit _deadman_ function. So why isn't it being used here?
https://docs.influxdata.com/flux/v0.x/stdlib/influxdata/influxdb/monitor/deadman/
Generated flux:
```javascript
import "strings"
import "regexp"
import "json"
import "influxdata/influxdb/secrets"
import "influxdata/influxdb/schema"
import "influxdata/influxdb/monitor"
import "http"
import "experimental"
option task = {name: "Notebook Deadman Task for local_8kauc-5uDcpMcxYZgTgdh", every: 5m, offset: 0s}
option v = {timeRangeStart: -15m, timeRangeStop: now()}
check = {
_check_id: "local_8kauc-5uDcpMcxYZgTgdh",
_check_name: "Notebook Generated Deadman Check",
_type: "deadman",
tags: {},
}
notification = {
_notification_rule_id: "local_8kauc-5uDcpMcxYZgTgdh",
_notification_rule_name: "Notebook Generated Rule",
_notification_endpoint_id: "local_8kauc-5uDcpMcxYZgTgdh",
_notification_endpoint_name: "Notebook Generated Endpoint",
}
task_data =
from(bucket: "devbucket")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "jsoucno")
|> filter(fn: (r) => r["_field"] == "mag")
trigger = (r) => r["dead"]
messageFn = (r) =>
"Deadman from NB
${strings.title(v: r._type)} for ${r._source_measurement} triggered at ${time(v: r._source_timestamp)}!"
task_data
|> schema["fieldsAsCols"]()
|> set(
key: "_notebook_link",
value: "https://twodotoh-dev-bonitoo-kk.remocal.influxdev.co/orgs/9c5955fc99a60b8f/notebooks/5fda5fc0020f0000",
)
|> monitor["check"](data: check, messageFn: messageFn, crit: trigger)
|> monitor["notify"](
data: notification,
endpoint:
http.endpoint(url: "http://ec2-13-52-255-26.us-west-1.compute.amazonaws.com:3000")(
mapFn: (r) => {
body = {r with _version: 1}
return {headers: {"Content-Type": "application/json"}, data: json.encode(v: body)}
},
),
)
|> monitor["deadman"](t: experimental["subDuration"](from: now(), d: 5m))
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.