influxdata / influxdata/kapacitor
Deadman OK Reports Incorrect Number of Points Emitted
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
OS: Linux version 4.1.10-202.fc22.x86_64 (Red Hat 5.3.1-2)
Kapacitor 1.2.0 (git: master 5408057e5a3493d3b5bd38d5d535ea45b587f8ff)
I have a deadman set up on a 10 minute timer. If the number of points seen falls below 100 then we trigger. The TICKscript looks like this:
```
TICKscript:
var s = stream
|from()
.database('sysQueries')
.retentionPolicy('autogen')
.measurement('fireMeasure')
.where(lambda: "hostBox" =~ /^(fireHost1|fireHost2)$/)
.groupBy('hostBox')
s
|deadman(100.0, 10m)
.id('fireAlert {{ .Group }}')
.message('{{ .ID }} is {{ .Level }}: {{ index .Fields "emitted" | printf "%.0f" }} points seen in last 10m')
.stateChangesOnly()
.log('/tmp/fireAlert.log')
.mode(0644)
```
We correctly triggered CRITICAL in a 10m interval and reset appropriately to OK in following 10m interval. However the numbers of points reported in the OK interval is inconsistent with the number we are actually seeing and recording in our influx db.
From fireAlert.log:
```
{"id":"fireAlert hostBox=fireHost1","message":"fireAlert hostBox=fireHost1 is CRITICAL: 76 points seen in last 10m","details":"{\u0026#34;Name\u0026#34;:\u0026#34;stats\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;fireAlert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;hostBox=fireHost1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;hostBox\u0026#34;:\u0026#34;fireHost1\u0026#34;,\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;emitted\u0026#34;:76},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2018-06-18T14:20:00Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1 is CRITICAL: 76 points seen in last 10m\u0026#34;}\n","time":"2018-06-18T14:20:00Z","duration":0,"level":"CRITICAL","data":{"Series":[{"name":"stats","tags":{"hostBox":"fireHost1"},"columns":["time","emitted"],"values":[["2018-06-18T14:20:00Z",76]]}],"Messages":null,"Err":null}}
{"id":"fireAlert hostBox=fireHost1","message":"fireAlert hostBox=fireHost1 is OK: 4622 points seen in last 10m","details":"{\u0026#34;Name\u0026#34;:\u0026#34;stats\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;fireAlert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;hostBox=fireHost1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;hostBox\u0026#34;:\u0026#34;fireHost1\u0026#34;,\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;emitted\u0026#34;:4622},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2018-06-18T14:30:00Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1 is OK: 4622 points seen in last 10m\u0026#34;}\n","time":"2018-06-18T14:30:00Z","duration":600000000000,"level":"OK","data":{"Series":[{"name":"stats","tags":{"hostBox":"fireHost1"},"columns":["time","emitted"],"values":[["2018-06-18T14:30:00Z",4622]]}],"Messages":null,"Err":null}}
```
The influx DB that has this data shows the correct numbers.
```
> select count(num) from fireMeasure where host =~ /^fireHost1$/ and time >= '2018-06-18T14:20:00Z' and time < '2018-06-18T14:40:00Z' group by time(10m)
name: fireMeasure
time count
---- -----
2018-06-18T14:20:00Z 76
2018-06-18T14:30:00Z 191
```
As you can see, the fireAlert.log shows CRITICAL and correctly reports 76 but OK incorrectly reports 4622.
Contributor guide
Research direction
Start with the deadman operator and reproduce the supplied TICKscript and interval counts, comparing the alert's emitted value with the InfluxDB results. Done means the OK alert reports the number of points emitted during its own 10-minute interval rather than the inflated value shown in the log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100