influxdata / influxdata/ui

Notebooks(Alerts): Deadman definition sends notifications even when stream is healthy

Open
#4,034 3 comments 0 reactions 0 assignees View on GitHub
area/notebooks bonitoo kind/bug team/automation team/ui team/unity
Dominant language
TypeScript
Stars
117
Forks
51
Avg merge
2d 15h
Merged PRs (30d)
4

Description

## About the bug

**Steps to reproduce:**
List the minimal actions needed to reproduce the behavior.

1. Setup an echo listener to receive HTTP messages
2. Go to Influxdbv2 UI (note that during this test the feature flag 'notebooksNewEndpoints' was set)
3. Add test data
4. Create an Alert Notebook
5. Using the test data define the query to retrieve a numeric stream
6. In the alert section define a trigger using the notification category "missing for longer than".
7. Set values for the time frame and cutoff that will fall within the test data (e.g. 5m timeframe and 15m cutoff).
8. Set the every value to match the time frame (e.g. 5m)
9. Select HTTP integration and use the URL for the listener in step 1
10. Export the notebook as an Alert Task.
11. Monitor messages sent to the listener

**Expected behavior:**

With a "deadman" alert, expect for notifications to arrive once the stream of test data comes to an end, and for only 1 notification to be sent per expiration during the 'every' period, given that in the time frame there are no points in the stream.

**Actual behavior:**

I'm seeing notifications with for example `_level: OK` coming from the created task. Also generating spam #3707.

I compared this with a classic confguration; deadman check, notification endpoint, notification rule, and this sent 1 notification when the data stream terminated and no notifications before then.

**Visual Proof:**

the definition
![NotebookDeadman01](https://user-images.githubusercontent.com/25881301/157253437-d40216e4-37a0-4c3f-b01d-32327fd951fa.png)

console messages from the listener
![NotebookDeadmanNotifications01](https://user-images.githubusercontent.com/25881301/157253507-a220f3ee-78ff-4859-979a-93a46c07e184.png)

console messages from the listener with standard deadman+endpoint+rule
![NotificationFromClassicDeadman01](https://user-images.githubusercontent.com/25881301/157253648-9f27c6cf-b39b-479c-9b77-6cb2070513c9.png)

## About your environment

**Environment info:**

Testing in K8SIDPE remocal

latest commmit in project UI

```
commit 22850c6b4a45fa63c8e8cf267439f90cc8e58460 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Author: Jakub Bednář
Date: Mon Mar 7 08:19:59 2022 +0100

```

latest commit in project K8SIDPE

```
commit e0ea3f9c0f3f5e186077e7cbe3aa399ba8abe317 (HEAD -> master, origin/master, origin/HEAD)
Author: wiedld
Date: Mon Mar 7 17:03:34 2022 -0800
```

## BTW

I see that the flux script for the generated task is using `|> monitor["check"](data: check, messageFn: messageFn, crit: trigger)`.

However the monitor api has an explicit _deadman_ function. So why isn't it being used here?
https://docs.influxdata.com/flux/v0.x/stdlib/influxdata/influxdb/monitor/deadman/

Generated flux:
```javascript
import "strings"
import "regexp"
import "json"
import "influxdata/influxdb/secrets"
import "influxdata/influxdb/schema"
import "influxdata/influxdb/monitor"
import "http"
import "experimental"

option task = {name: "Notebook Deadman Task for local_8kauc-5uDcpMcxYZgTgdh", every: 5m, offset: 0s}
option v = {timeRangeStart: -15m, timeRangeStop: now()}

check = {
_check_id: "local_8kauc-5uDcpMcxYZgTgdh",
_check_name: "Notebook Generated Deadman Check",
_type: "deadman",
tags: {},
}

notification = {
_notification_rule_id: "local_8kauc-5uDcpMcxYZgTgdh",
_notification_rule_name: "Notebook Generated Rule",
_notification_endpoint_id: "local_8kauc-5uDcpMcxYZgTgdh",
_notification_endpoint_name: "Notebook Generated Endpoint",
}

task_data =
from(bucket: "devbucket")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "jsoucno")
|> filter(fn: (r) => r["_field"] == "mag")
trigger = (r) => r["dead"]
messageFn = (r) =>
"Deadman from NB
${strings.title(v: r._type)} for ${r._source_measurement} triggered at ${time(v: r._source_timestamp)}!"

task_data
|> schema["fieldsAsCols"]()
|> set(
key: "_notebook_link",
value: "https://twodotoh-dev-bonitoo-kk.remocal.influxdev.co/orgs/9c5955fc99a60b8f/notebooks/5fda5fc0020f0000",
)
|> monitor["check"](data: check, messageFn: messageFn, crit: trigger)
|> monitor["notify"](
data: notification,
endpoint:
http.endpoint(url: "http://ec2-13-52-255-26.us-west-1.compute.amazonaws.com:3000")(
mapFn: (r) => {
body = {r with _version: 1}

return {headers: {"Content-Type": "application/json"}, data: json.encode(v: body)}
},
),
)
|> monitor["deadman"](t: experimental["subDuration"](from: now(), d: 5m))
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.