influxdata / influxdata/kapacitor
Using .noRecoveries() disables level reset of alert
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
### Description
According to [documentation](https://docs.influxdata.com/kapacitor/v1.6/nodes/alert_node/#norecoveries) `.noRecoveries()` on alert disables sending recovery alerts and sets `recoverable` alert data field to `false`. However it seems that it totally disables resetting the level of the alert and not just disabling sending recovery alerts.
### What is happening
(Database and Retention Policy are masked)
Having a kapacitor task similar based on the following template :
```
var application string
var message = '{{ index .Tags "node" }} of {{ index .Tags "application" }} is {{ index .Fields "up" }}!'
batch
|query('SELECT "up" FROM "".""."status" WHERE "application"= \'' + application + '\'')
.groupBy('application','node')
.period(10s)
.every(10s)
|log()
|alert()
.id('Status {{ index .Tags "application" }}-{{ index .Tags "node" }}')
.crit(lambda: "up" == FALSE)
.message(message)
.noRecoveries()
.teams()
.log('/dev/stdout')
```
The following log segment is produced for a data scenario ```CRITICAL->OK->CRITICAL``` :
```
ts=2022-04-05T05:53:27.570Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T05:53:17.570443921Z' AND time < '2022-04-05T05:53:27.570443921Z' GROUP BY application, node"
ts=2022-04-05T05:53:27.574Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:27.570443921Z
ts=2022-04-05T05:53:27.574Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=false time=2022-04-05T05:53:24.0323641Z
ts=2022-04-05T05:53:27.574Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:27.570443921Z
ts=2022-04-05T05:53:27.575Z lvl=debug msg="alert triggered" service=kapacitor task_master=main task=acme-up-alert-task node=alert3 level=CRITICAL id="Status acme-node1" event_message="node1 of acme is false!" data="&{status map[application:acme node:node1] [time up] [[2022-04-05 05:53:24.0323641 +0000 UTC false]]}"
{"id":"Status acme-node1","message":"node1 of acme is false!","details":"{\u0026#34;Name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;acme-up-alert-task\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;application=acme,node=node1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;application\u0026#34;:\u0026#34;acme\u0026#34;,\u0026#34;node\u0026#34;:\u0026#34;node1\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor.mlqr-test\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;8f56376a-c975-4ea0-83c5-16ba71f40fce\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ab3e86de-78d9-452e-8a97-38a98a6b2833\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;Status acme-node1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;up\u0026#34;:false},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2022-04-05T05:53:24.0323641Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;node1 of acme is false!\u0026#34;}\n","time":"2022-04-05T05:53:24.0323641Z","duration":0,"level":"CRITICAL","data":{"series":[{"name":"status","tags":{"application":"acme","node":"node1"},"columns":["time","up"],"values":[["2022-04-05T05:53:24.0323641Z",false]]}]},"previousLevel":"OK","recoverable":false}
ts=2022-04-05T05:53:37.570Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T05:53:27.570212103Z' AND time < '2022-04-05T05:53:37.570212103Z' GROUP BY application, node"
ts=2022-04-05T05:53:47.570Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T05:53:37.570425599Z' AND time < '2022-04-05T05:53:47.570425599Z' GROUP BY application, node"
ts=2022-04-05T05:53:47.573Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:47.570425599Z
ts=2022-04-05T05:53:47.573Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=true time=2022-04-05T05:53:39.3161006Z
ts=2022-04-05T05:53:47.573Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:47.570425599Z
ts=2022-04-05T05:53:57.570Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T05:53:47.57017437Z' AND time < '2022-04-05T05:53:57.57017437Z' GROUP BY application, node"
ts=2022-04-05T05:53:57.574Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:57.57017437Z
ts=2022-04-05T05:53:57.574Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=false time=2022-04-05T05:53:54.7846331Z
ts=2022-04-05T05:53:57.574Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:57.57017437Z
ts=2022-04-05T05:53:57.574Z lvl=debug msg="alert triggered" service=kapacitor task_master=main task=acme-up-alert-task node=alert3 level=CRITICAL id="Status acme-node1" event_message="node1 of acme is false!" data="&{status map[application:acme node:node1] [time up] [[2022-04-05 05:53:54.7846331 +0000 UTC false]]}"
{"id":"Status acme-node1","message":"node1 of acme is false!","details":"{\u0026#34;Name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;acme-up-alert-task\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;application=acme,node=node1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;application\u0026#34;:\u0026#34;acme\u0026#34;,\u0026#34;node\u0026#34;:\u0026#34;node1\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor.mlqr-test\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;8f56376a-c975-4ea0-83c5-16ba71f40fce\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ab3e86de-78d9-452e-8a97-38a98a6b2833\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;Status acme-node1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;up\u0026#34;:false},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2022-04-05T05:53:54.7846331Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;node1 of acme is false!\u0026#34;}\n","time":"2022-04-05T05:53:54.7846331Z","duration":0,"level":"CRITICAL","data":{"series":[{"name":"status","tags":{"application":"acme","node":"node1"},"columns":["time","up"],"values":[["2022-04-05T05:53:54.7846331Z",false]]}]},"previousLevel":"CRITICAL","recoverable":false}
ts=2022-04-05T05:54:07.570Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T05:53:57.57023309Z' AND time < '2022-04-05T05:54:07.57023309Z' GROUP BY application, node"
```
When the first batch with "Critical" data is received (based on the output of the ```|log()``` node):
```
ts=2022-04-05T05:53:27.574Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:27.570443921Z
ts=2022-04-05T05:53:27.574Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=false time=2022-04-05T05:53:24.0323641Z
ts=2022-04-05T05:53:27.574Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:27.570443921Z
```
an alert is triggered and in the alert data:
```
"previousLevel":"OK"
```
as expected.
Afterwards the second batch with "OK" data is received:
```
ts=2022-04-05T05:53:47.573Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:47.570425599Z
ts=2022-04-05T05:53:47.573Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=true time=2022-04-05T05:53:39.3161006Z
ts=2022-04-05T05:53:47.573Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:47.570425599Z
```
No alert is triggered because of ```.noRecoveries()``` but I would expect that the level of the alert is reset.
When the third batch with "Critical" data is received:
```
ts=2022-04-05T05:53:57.574Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:57.57017437Z
ts=2022-04-05T05:53:57.574Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=false time=2022-04-05T05:53:54.7846331Z
ts=2022-04-05T05:53:57.574Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T05:53:57.57017437Z
```
an alert is triggered but in the alert data:
```
"previousLevel":"CRITICAL"
```
### What was expected
The second alert (generated by the third batch) should have:
```
"previousLevel":"OK"
```
since the data in the second batch is capable of resetting the alert. This can be seen when ```|noRecoveries()``` is removed and the following log segment is produced:
```
ts=2022-04-05T06:18:41.683Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T06:18:41.680749198Z
ts=2022-04-05T06:18:41.684Z lvl=debug msg="alert triggered" service=kapacitor task_master=main task=acme-up-alert-task node=alert3 level=CRITICAL id="Status acme-node1" event_message="node1 of acme is false!" data="&{status map[application:acme node:node1] [time up] [[2022-04-05 06:18:35.9500652 +0000 UTC false]]}"
{"id":"Status acme-node1","message":"node1 of acme is false!","details":"{\u0026#34;Name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;acme-up-alert-task\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;application=acme,node=node1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;application\u0026#34;:\u0026#34;acme\u0026#34;,\u0026#34;node\u0026#34;:\u0026#34;node1\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor.mlqr-test\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;51fcf587-7062-4a0c-b63c-9434bca60841\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;5fabbeae-0286-4366-a633-4fcc3f417458\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;Status acme-node1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;up\u0026#34;:false},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2022-04-05T06:18:35.9500652Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;node1 of acme is false!\u0026#34;}\n","time":"2022-04-05T06:18:35.9500652Z","duration":0,"level":"CRITICAL","data":{"series":[{"name":"status","tags":{"application":"acme","node":"node1"},"columns":["time","up"],"values":[["2022-04-05T06:18:35.9500652Z",false]]}]},"previousLevel":"OK","recoverable":true}
ts=2022-04-05T06:18:51.681Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T06:18:41.681653263Z' AND time < '2022-04-05T06:18:51.681653263Z' GROUP BY application, node"
ts=2022-04-05T06:18:51.685Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_node=node1 tag_application=acme time=2022-04-05T06:18:51.681653263Z
ts=2022-04-05T06:18:51.685Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=true time=2022-04-05T06:18:49.8301689Z
ts=2022-04-05T06:18:51.685Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T06:18:51.681653263Z
ts=2022-04-05T06:18:51.685Z lvl=debug msg="alert triggered" service=kapacitor task_master=main task=acme-up-alert-task node=alert3 level=OK id="Status acme-node1" event_message="node1 of acme is true!" data="&{status map[application:acme node:node1] [time up] [[2022-04-05 06:18:49.8301689 +0000 UTC true]]}"
{"id":"Status acme-node1","message":"node1 of acme is true!","details":"{\u0026#34;Name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;acme-up-alert-task\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;application=acme,node=node1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;application\u0026#34;:\u0026#34;acme\u0026#34;,\u0026#34;node\u0026#34;:\u0026#34;node1\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor.mlqr-test\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;51fcf587-7062-4a0c-b63c-9434bca60841\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;5fabbeae-0286-4366-a633-4fcc3f417458\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;Status acme-node1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;up\u0026#34;:true},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2022-04-05T06:18:51.681653263Z\u0026#34;,\u0026#34;Duration\u0026#34;:15731588063,\u0026#34;Message\u0026#34;:\u0026#34;node1 of acme is true!\u0026#34;}\n","time":"2022-04-05T06:18:51.681653263Z","duration":15731588063,"level":"OK","data":{"series":[{"name":"status","tags":{"application":"acme","node":"node1"},"columns":["time","up"],"values":[["2022-04-05T06:18:49.8301689Z",true]]}]},"previousLevel":"CRITICAL","recoverable":true}
ts=2022-04-05T06:19:01.681Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T06:18:51.6811473Z' AND time < '2022-04-05T06:19:01.6811473Z' GROUP BY application, node"
ts=2022-04-05T06:19:11.681Z lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=acme-up-alert-task node=query1 query="SELECT up FROM \"\"..status WHERE application = 'acme' AND time >= '2022-04-05T06:19:01.681747154Z' AND time < '2022-04-05T06:19:11.681747154Z' GROUP BY application, node"
ts=2022-04-05T06:19:11.685Z lvl=info msg="begin batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T06:19:11.681747154Z
ts=2022-04-05T06:19:11.685Z lvl=info msg="batch point" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 field_up=false time=2022-04-05T06:19:07.1216388Z
ts=2022-04-05T06:19:11.685Z lvl=info msg="end batch" service=kapacitor task_master=main task=acme-up-alert-task node=log2 prefix= name=status group=application=acme,node=node1 tag_application=acme tag_node=node1 time=2022-04-05T06:19:11.681747154Z
ts=2022-04-05T06:19:11.685Z lvl=debug msg="alert triggered" service=kapacitor task_master=main task=acme-up-alert-task node=alert3 level=CRITICAL id="Status acme-node1" event_message="node1 of acme is false!" data="&{status map[application:acme node:node1] [time up] [[2022-04-05 06:19:07.1216388 +0000 UTC false]]}"
{"id":"Status acme-node1","message":"node1 of acme is false!","details":"{\u0026#34;Name\u0026#34;:\u0026#34;status\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;acme-up-alert-task\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;application=acme,node=node1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;application\u0026#34;:\u0026#34;acme\u0026#34;,\u0026#34;node\u0026#34;:\u0026#34;node1\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor.mlqr-test\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;51fcf587-7062-4a0c-b63c-9434bca60841\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;5fabbeae-0286-4366-a633-4fcc3f417458\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;Status acme-node1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;up\u0026#34;:false},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2022-04-05T06:19:07.1216388Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;node1 of acme is false!\u0026#34;}\n","time":"2022-04-05T06:19:07.1216388Z","duration":0,"level":"CRITICAL","data":{"series":[{"name":"status","tags":{"application":"acme","node":"node1"},"columns":["time","up"],"values":[["2022-04-05T06:19:07.1216388Z",false]]}]},"previousLevel":"OK","recoverable":true}
```
According to documentation the ```.noRecoveries()``` should disable sending recovery alerts, but based on the fact that the second alert has ```previousLevel":"CRITICAL```, it seems that ```.noRecoveries()``` disables level reset of an alert.
### Setup Details
- Kapacitor Version: 1.6.3
- Installed using a pre-built [RPM package](https://dl.influxdata.com/kapacitor/releases/kapacitor-1.6.3-1.x86_64.rpm) in a custom container image based on [Red Hat Universal Base Image 8](https://catalog.redhat.com/software/containers/ubi8/ubi/5c359854d70cc534b3a3784e)
- Deployed on a Kubernetes cluster using this [helm chart](https://github.com/influxdata/helm-charts/tree/master/charts/kapacitor)
Contributor guide
Research direction
Start with the alert node's handling of .noRecoveries() and reproduce the reported CRITICAL→OK→CRITICAL sequence from the task template. Compare alert data, especially previousLevel and recoverable, with and without .noRecoveries(). Done means recovery notifications remain suppressed while the alert level resets to OK before a later critical event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100