influxdata / influxdata/kapacitor
Too many alerts even with stateChangesOnly - bug in evalNode?
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi!
I'm using Kapacitor 1.3 and InfluxDB 1.2.
I use an influxDBOutNode to write in influxDB. So as not to get spam all day long I use a stateChangesOnly(24h) on my alertNode.
Yet sometimes (not always, I didn't manage to find a criteria) multiple alerts are written on influxDB. Plus, values of fields are modified where they souldn't have been (I suspect due to an evalNode).
Here is my tick script:
```
var alert = data
|alert()
.crit(lambda: "value"!=0 AND "value"!=1)
.message('ENERWHERE {{ .Level }} ALERT: anormal value of drive1_mode = {{index .Fields "value"}} (should be 0 or 1)')
.id('driveMode')
.idTag('id')
.levelTag('level')
.messageField('message')
// Alert
alert
.log('/tmp/driveMode_alert.log')
//Writing on InfluxDB
|eval(lambda: if ("value"!=0 AND "value"!=1, 1, 0), lambda: "value", lambda: "message")
.as('crit','data','msg')
.keep('crit','data','msg')
|where(lambda: "id" != '')
|influxDBOut()
.database('enerwhere')
.retentionPolicy('autogen')
.measurement('alert')
.tag('tracker',tracker)
```
Here is the data I find strange:
```
InfluxDB shell version: 1.2.4
> select crit,id,level,data,msg from alert where id='driveMode'
name: alert
time crit id level data msg
---- ---- -- ----- ---- ---
1497174631000000000 1 driveMode CRITICAL 4 ENERWHERE CRITICAL ALERT : anormal value of drive1_mode = 4 (should be 0 or 1)
1497174632000000000 0 driveMode CRITICAL 1 ENERWHERE CRITICAL ALERT : anormal value of drive1_mode = 4 (should be 0 or 1)
1497193780000000000 0 driveMode OK 0 ENERWHERE OK ALERT : anormal value of drive1_mode = 0 (should be 0 or 1)
```
First, I don't understand why two ```CRITICAL``` alerts are written in influxDB. I verified, only one has been logged on ```/tmp/driveMode_alert.log```.
Second, as you can see in the script, when the ```level``` is ```CRTITICAL``` the field ```crit``` should be 1. In addition, the field ```data``` is nothing but the field ```value``` renamed, ```data``` has changed but the ```msg``` is identical!
Any idea?
Contributor guide
Research direction
Start by reproducing the issue with the provided TICK script, focusing on the alert(), stateChangesOnly(24h), eval(), and influxDBOut() stages. Compare the alert log with the InfluxDB query results, then trace evalNode behavior and add a regression test covering duplicate writes and inconsistent fields. Done means the alert is written once and its fields remain consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100